May 20 13:47:42.795212 ip-10-0-139-248 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 20 13:47:42.795224 ip-10-0-139-248 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 20 13:47:42.795233 ip-10-0-139-248 systemd[1]: kubelet.service: Failed with result 'resources'.
May 20 13:47:42.795549 ip-10-0-139-248 systemd[1]: Failed to start Kubernetes Kubelet.
May 20 13:47:52.966244 ip-10-0-139-248 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 20 13:47:52.966260 ip-10-0-139-248 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 356da0962db14d9e85d80c07f4584223 --
May 20 13:50:19.753140 ip-10-0-139-248 systemd[1]: Starting Kubernetes Kubelet...
May 20 13:50:20.162267 ip-10-0-139-248 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:20.162267 ip-10-0-139-248 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 13:50:20.162267 ip-10-0-139-248 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:20.162267 ip-10-0-139-248 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 13:50:20.162751 ip-10-0-139-248 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:20.163090 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.163022    2557 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 13:50:20.165721 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165701    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:20.165721 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165720    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:20.165721 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165726    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165731    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165736    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165740    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165744    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165748    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165751    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165755    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165759    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165763    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165767    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165772    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165776    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165781    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165785    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165789    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165793    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165797    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165802    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165806    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:20.165927 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165810    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165814    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165818    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165822    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165827    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165831    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165835    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165839    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165844    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165848    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165870    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165877    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165882    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165886    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165891    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165895    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165900    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165904    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165909    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:20.166732 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165914    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165918    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165922    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165926    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165930    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165934    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165938    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165943    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165947    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165951    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165956    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165960    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165964    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165969    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165974    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165979    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165983    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165988    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165992    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.165996    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:20.167556 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166000    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166004    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166009    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166013    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166017    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166022    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166026    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166032    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166036    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166040    2557 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166045    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166049    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166068    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166074    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166078    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166083    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166087    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166098    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166103    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166108    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:20.168400 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166112    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166116    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166120    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166124    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166128    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166703    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166712    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166716    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166721    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166726    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166729    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166733    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166737    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166741    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166745    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166749    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166753    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166758    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166762    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166767    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:20.169026 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166771    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166775    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166780    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166784    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166789    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166794    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166798    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166802    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166807    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166811    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166815    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166820    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166824    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166828    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166832    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166837    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166841    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166845    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166871    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166876    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:20.169583 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166880    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166886    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166890    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166894    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166899    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166904    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166908    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166912    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166916    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166920    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166924    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166929    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166933    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166937    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166942    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166946    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166950    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166955    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166959    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:20.170321 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166964    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166969    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166973    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166977    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166981    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166985    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166989    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.166994    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167001    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167007    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167012    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167017    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167024    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167029    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167034    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167039    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167043    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167047    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:20.170906 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167052    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167056    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167060    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167064    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167068    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167072    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167077    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167081    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167085    2557 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167089    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167094    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167098    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167102    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.167114    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167217    2557 flags.go:64] FLAG: --address="0.0.0.0"
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167227    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167238    2557 flags.go:64] FLAG: --anonymous-auth="true"
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167245    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167255    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167260    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167266    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 13:50:20.171427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167274    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167279    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167284    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167290    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167295    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167300    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167306    2557 flags.go:64] FLAG: --cgroup-root=""
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167311    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167316    2557 flags.go:64] FLAG: --client-ca-file=""
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167321    2557 flags.go:64] FLAG: --cloud-config=""
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167326    2557 flags.go:64] FLAG: --cloud-provider="external"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167331    2557 flags.go:64] FLAG: --cluster-dns="[]"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167340    2557 flags.go:64] FLAG: --cluster-domain=""
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167345    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167350    2557 flags.go:64] FLAG: --config-dir=""
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167355    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167360    2557 flags.go:64] FLAG: --container-log-max-files="5"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167367    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167372    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167377    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167382    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167388    2557 flags.go:64] FLAG: --contention-profiling="false"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167393    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167398    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167403    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 13:50:20.171966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167415    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167423    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167428    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167433    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167440    2557 flags.go:64] FLAG: --enable-load-reader="false"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167446    2557 flags.go:64] FLAG: --enable-server="true"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167451    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167461    2557 flags.go:64] FLAG: --event-burst="100"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167466    2557 flags.go:64] FLAG: --event-qps="50"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167471    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167477    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167482    2557 flags.go:64] FLAG: --eviction-hard=""
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167488    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167493    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167499    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167504    2557 flags.go:64] FLAG: --eviction-soft=""
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167510    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167514    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167519    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167525    2557 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167530    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167535    2557 flags.go:64] FLAG: --fail-swap-on="true"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167540    2557 flags.go:64] FLAG: --feature-gates=""
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167546    2557 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167551    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 13:50:20.172591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167557    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167562    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167567    2557 flags.go:64] FLAG: --healthz-port="10248"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167572    2557 flags.go:64] FLAG: --help="false"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167577    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-139-248.ec2.internal"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167582    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167587    2557 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167592    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167600    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167606    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167610    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167617    2557 flags.go:64] FLAG: --image-service-endpoint=""
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167622    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167627    2557 flags.go:64] FLAG: --kube-api-burst="100"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167632    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167638    2557 flags.go:64] FLAG: --kube-api-qps="50"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167643    2557 flags.go:64] FLAG: --kube-reserved=""
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167648    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167653    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167658    2557 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167663    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167668    2557 flags.go:64] FLAG: --lock-file=""
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167674    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167679    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 13:50:20.173209 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167684    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167693    2557 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167698    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167703    2557 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167708    2557 flags.go:64] FLAG: --logging-format="text"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167713    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167718    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167724    2557 flags.go:64] FLAG: --manifest-url=""
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167729    2557 flags.go:64] FLAG: --manifest-url-header=""
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167736    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167741    2557 flags.go:64] FLAG: --max-open-files="1000000"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167747    2557 flags.go:64] FLAG: --max-pods="110"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167752    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167757    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167762    2557 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167767    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167772    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167782    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167788    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167801    2557 flags.go:64] FLAG: --node-status-max-images="50"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167807    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167813    2557 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167818    2557 flags.go:64] FLAG: --pod-cidr=""
May 20 13:50:20.173822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167822    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167831    2557 flags.go:64] FLAG: --pod-manifest-path=""
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167837    2557 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167842    2557 flags.go:64] FLAG: --pods-per-core="0"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167846    2557 flags.go:64] FLAG: --port="10250"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167868    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167873    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0331132e7025a7e4d"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167882    2557 flags.go:64] FLAG: --qos-reserved=""
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167888    2557 flags.go:64] FLAG: --read-only-port="10255"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167893    2557 flags.go:64] FLAG: --register-node="true"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167898    2557 flags.go:64] FLAG: --register-schedulable="true"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167902    2557 flags.go:64] FLAG: --register-with-taints=""
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167909    2557 flags.go:64] FLAG: --registry-burst="10"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167914    2557 flags.go:64] FLAG: --registry-qps="5"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167919    2557 flags.go:64] FLAG: --reserved-cpus=""
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167923    2557 flags.go:64] FLAG: --reserved-memory=""
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167930    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167935    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167940    2557 flags.go:64] FLAG: --rotate-certificates="false"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167946    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167950    2557 flags.go:64] FLAG: --runonce="false"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167955    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167962    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167968    2557 flags.go:64] FLAG: --seccomp-default="false"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167973    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167978    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 13:50:20.174400 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167983    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.167994    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168002    2557 flags.go:64] FLAG: --storage-driver-password="root"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168007    2557 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168013    2557 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168017    2557 flags.go:64] FLAG: --storage-driver-user="root"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168023    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168028    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168033    2557 flags.go:64] FLAG: --system-cgroups=""
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168038    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168047    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168052    2557 flags.go:64] FLAG: --tls-cert-file=""
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168057    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168066    2557 flags.go:64] FLAG: --tls-min-version=""
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168071    2557 flags.go:64] FLAG: --tls-private-key-file=""
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168076    2557 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168081    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168086    2557 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168091    2557 flags.go:64] FLAG: --v="2"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168098    2557 flags.go:64] FLAG: --version="false"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168105    2557 flags.go:64] FLAG: --vmodule=""
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168111    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.168117    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168265    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:20.175019 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168271    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168276    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168281    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168285    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168290    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168294    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168299    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168303    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168308    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168312    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168319    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168324    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168329    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168334    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168339    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168343    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168348    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168352    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168356    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168360    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:20.175800 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168364    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168370    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168375    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168379    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168384    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168388    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168392    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168396    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168401    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168405    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168409    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168413    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168417    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168421    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168425    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168429    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168434    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168440    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168446    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:20.176345 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168451    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168455    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168459    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168464    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168470    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168475    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168479    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168483    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168488    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168492    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168496    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168500    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168505    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168509    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168514    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168520    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168524    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168528    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168533    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168537    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:20.176811 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168541    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168545    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168550    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168554    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168558    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168563    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168570    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168575    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168580    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168584    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168588    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168593    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168597    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168601    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168606    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168610    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168615    2557 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168620    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168624    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:20.177303 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168628    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168633    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168637    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168641    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168646    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168650    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.168654    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.169199    2557 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.177231    2557 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.177356    2557 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177426    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177434    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177439    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177443    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177449    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177453    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:20.177770 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177456    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177461    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177467    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177473    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177478    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177482    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177486    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177490    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177495    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177499    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177503    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177507    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177511    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177515    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177519    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177523    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177527    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177531    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177536    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:20.178185 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177540    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177543    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177546    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177548    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177551    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177554    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177557    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177560    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177563    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177566    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177568    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177571    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177573    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177576    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177578    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177581    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177583    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177586    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177588    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177591    2557 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:20.178634 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177593    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177595    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177598    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177600    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177603    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177605    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177608    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177610    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177613    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177615    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177619    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177621    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177624    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177626    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177628    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177631    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177633    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177636    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177640    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177642    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:20.179126 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177645    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177649    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177653    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177656    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177658    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177661    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177663    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177666    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177669    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177671    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177673    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177676    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177678    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177681    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177683    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177686    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177688    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177691    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177694    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177696    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:20.179599 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177699    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.177704    2557 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177790    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177795    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177798    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177801    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177803    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177806    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177809    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177811    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177814    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177817    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177820    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177822    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177825    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177827    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:20.180156 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177830    2557 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177833    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177835    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177838    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177840    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177843    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177845    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177848    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177868    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177873    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177877    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177881    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177884    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177887    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177889    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177892    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177894    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177897    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177900    2557 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177903    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:20.180538 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177905    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177908    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177910    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177913    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177915    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177919    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177924    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177927    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177930    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177932    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177935    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177938    2557 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177941    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177943    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177946    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177948    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177951    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177953    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177956    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:20.181028 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177958    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177960    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177964    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177966    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177969    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177972    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177974    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177977    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177979    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177982    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177984    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177987    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177989    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177991    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177994    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177996    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.177999    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178002    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178004    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178006    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:20.181505 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178009    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178011    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178014    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178016    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178019    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178021    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178023    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178026    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178029    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178032    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178035    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178037    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:20.178040    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.178044    2557 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:20.181995 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.178841    2557 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 13:50:20.184154 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.184141    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 13:50:20.185229 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.185218    2557 server.go:1019] "Starting client certificate rotation"
May 20 13:50:20.185325 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.185313    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:20.185357 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.185344    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:20.206971 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.206955    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:20.210516 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.210502    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:20.221342 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.221325    2557 log.go:25] "Validated CRI v1 runtime API"
May 20 13:50:20.227100 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.227088    2557 log.go:25] "Validated CRI v1 image API"
May 20 13:50:20.228288 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.228268    2557 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 13:50:20.231499 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.231481    2557 fs.go:135] Filesystem UUIDs: map[3bcd2c5c-f3a3-4b53-8e93-382022f907a6:/dev/nvme0n1p4 6b68edb8-82c2-4dbd-b4c5-ce9e257312a0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
May 20 13:50:20.231569 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.231497    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 13:50:20.236053 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.236037    2557 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:20.236339 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.236240    2557 manager.go:217] Machine: {Timestamp:2026-05-20 13:50:20.235737327 +0000 UTC m=+0.373291538 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198748 MemoryCapacity:33164505088 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2709544b759608e54181ac3f580ac1 SystemUUID:ec270954-4b75-9608-e541-81ac3f580ac1 BootID:356da096-2db1-4d9e-85d8-0c07f4584223 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582254592 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c6:f4:7d:56:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c6:f4:7d:56:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:58:05:20:b9:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164505088 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 13:50:20.236339 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.236339    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 13:50:20.236424 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.236418    2557 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 13:50:20.240200 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.240007    2557 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 13:50:20.240375 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.240207    2557 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 13:50:20.240427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.240387    2557 topology_manager.go:138] "Creating topology manager with none policy"
May 20 13:50:20.240427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.240396    2557 container_manager_linux.go:306] "Creating device plugin manager"
May 20 13:50:20.240427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.240404    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:20.240427 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.240418    2557 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:20.242185 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.242174    2557 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:20.242297 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.242288    2557 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 13:50:20.244884 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.244873    2557 kubelet.go:491] "Attempting to sync node with API server"
May 20 13:50:20.244917 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.244889    2557 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 13:50:20.245644 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.245634    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 13:50:20.245681 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.245648    2557 kubelet.go:397] "Adding apiserver pod source"
May 20 13:50:20.245681 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.245656    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 13:50:20.246612 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.246600    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:20.246658 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.246619    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:20.249570 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.249556    2557 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 13:50:20.251307 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.251293    2557 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 13:50:20.252452 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252439    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 13:50:20.252491 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252460    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 13:50:20.252491 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252470    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 13:50:20.252491 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252479    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 13:50:20.252491 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252490    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 13:50:20.252595 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252499    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 13:50:20.252595 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252507    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 13:50:20.252595 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252515    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 13:50:20.252595 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252525    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 13:50:20.252595 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252533    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 13:50:20.252595 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252545    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 13:50:20.252781 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.252770    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 13:50:20.253589 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.253579    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 13:50:20.253626 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.253590    2557 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 13:50:20.255967 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.255943    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 20 13:50:20.256038 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.255962    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 20 13:50:20.256399 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.256386    2557 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 13:50:20.256435 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.256426    2557 server.go:1295] "Started kubelet"
May 20 13:50:20.256533 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.256504    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 13:50:20.256606 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.256492    2557 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 13:50:20.256679 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.256623    2557 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 13:50:20.257010 ip-10-0-139-248 systemd[1]: Started Kubernetes Kubelet.
May 20 13:50:20.259039 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.259025    2557 server.go:317] "Adding debug handlers to kubelet server"
May 20 13:50:20.259973 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.259959    2557 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 13:50:20.262750 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.262734    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gxz2s"
May 20 13:50:20.264381 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.264368    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 13:50:20.264903 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.264889    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 13:50:20.265041 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.265023    2557 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 13:50:20.265041 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.265041    2557 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 13:50:20.265124 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.265073    2557 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 13:50:20.265178 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.265137    2557 reconstruct.go:97] "Volume reconstruction finished"
May 20 13:50:20.265178 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.265145    2557 reconciler.go:26] "Reconciler: start to sync state"
May 20 13:50:20.265666 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.265646    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:20.267479 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.267457    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 13:50:20.267479 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.267480    2557 factory.go:55] Registering systemd factory
May 20 13:50:20.267614 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.267491    2557 factory.go:223] Registration of the systemd container factory successfully
May 20 13:50:20.267721 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.267706    2557 factory.go:153] Registering CRI-O factory
May 20 13:50:20.267787 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.267725    2557 factory.go:223] Registration of the crio container factory successfully
May 20 13:50:20.267787 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.267742    2557 factory.go:103] Registering Raw factory
May 20 13:50:20.267787 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.267782    2557 manager.go:1196] Started watching for new ooms in manager
May 20 13:50:20.268151 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.268135    2557 manager.go:319] Starting recovery of all containers
May 20 13:50:20.269439 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.269417    2557 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 20 13:50:20.270309 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.269461    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-248.ec2.internal.18b14a4c828e7168  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-248.ec2.internal,UID:ip-10-0-139-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-248.ec2.internal,},FirstTimestamp:2026-05-20 13:50:20.25639972 +0000 UTC m=+0.393953928,LastTimestamp:2026-05-20 13:50:20.25639972 +0000 UTC m=+0.393953928,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-248.ec2.internal,}"
May 20 13:50:20.270604 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.270578    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 20 13:50:20.270768 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.270723    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 20 13:50:20.270986 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.270967    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gxz2s"
May 20 13:50:20.273210 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.273191    2557 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 13:50:20.281704 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.281689    2557 manager.go:324] Recovery completed
May 20 13:50:20.286024 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.286010    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:20.287635 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.287617    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:20.287727 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.287647    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:20.287727 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.287661    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:20.288167 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.288152    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 13:50:20.288167 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.288164    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 13:50:20.288277 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.288178    2557 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:20.290452 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.290441    2557 policy_none.go:49] "None policy: Start"
May 20 13:50:20.290496 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.290456    2557 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 13:50:20.290496 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.290466    2557 state_mem.go:35] "Initializing new in-memory state store"
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.327452    2557 manager.go:341] "Starting Device Plugin manager"
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.327490    2557 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.327498    2557 server.go:85] "Starting device plugin registration server"
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.327681    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.327690    2557 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.327845    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.327946    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.327957    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.328484    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 13:50:20.339246 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.328511    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:20.399135 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.399106    2557 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 13:50:20.400223 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.400207    2557 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 13:50:20.400282 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.400232    2557 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 13:50:20.400282 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.400251    2557 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 13:50:20.400282 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.400259    2557 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 13:50:20.400401 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.400299    2557 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 13:50:20.403985 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.403969    2557 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:20.428771 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.428739    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:20.429537 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.429522    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:20.429600 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.429551    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:20.429600 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.429563    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:20.429600 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.429582    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-248.ec2.internal"
May 20 13:50:20.437503 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.437490    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-248.ec2.internal"
May 20 13:50:20.437556 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.437509    2557 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-248.ec2.internal\": node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:20.456785 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.456764    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:20.500736 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.500715    2557 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"]
May 20 13:50:20.500808 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.500766    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:20.501504 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.501489    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:20.501558 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.501517    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:20.501558 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.501527    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:20.502780 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.502767    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:20.502979 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.502966    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.503017 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.502997    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:20.503426 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.503403    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:20.503529 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.503439    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:20.503529 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.503452    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:20.503529 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.503461    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:20.503529 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.503479    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:20.503529 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.503488    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:20.504580 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.504567    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.504642 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.504591    2557 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:20.505230 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.505212    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:20.505300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.505244    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:20.505300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.505260    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:20.533377 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.533353    2557 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-248.ec2.internal\" not found" node="ip-10-0-139-248.ec2.internal"
May 20 13:50:20.537428 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.537412    2557 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-248.ec2.internal\" not found" node="ip-10-0-139-248.ec2.internal"
May 20 13:50:20.557127 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.557109    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:20.567429 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.567410    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c85c3e656afa7121233f89f52297b573-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"c85c3e656afa7121233f89f52297b573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.567475 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.567432    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7bb2fc9c4d251ff13095ab5d1d57128-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.ec2.internal\" (UID: \"c7bb2fc9c4d251ff13095ab5d1d57128\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.567475 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.567448    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c85c3e656afa7121233f89f52297b573-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"c85c3e656afa7121233f89f52297b573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.657819 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.657799    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:20.668139 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.668119    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7bb2fc9c4d251ff13095ab5d1d57128-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.ec2.internal\" (UID: \"c7bb2fc9c4d251ff13095ab5d1d57128\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.668186 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.668148    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c85c3e656afa7121233f89f52297b573-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"c85c3e656afa7121233f89f52297b573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.668186 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.668165    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c85c3e656afa7121233f89f52297b573-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"c85c3e656afa7121233f89f52297b573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.668245 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.668198    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c85c3e656afa7121233f89f52297b573-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"c85c3e656afa7121233f89f52297b573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.668245 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.668216    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c7bb2fc9c4d251ff13095ab5d1d57128-config\") pod \"kube-apiserver-proxy-ip-10-0-139-248.ec2.internal\" (UID: \"c7bb2fc9c4d251ff13095ab5d1d57128\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.668245 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.668221    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c85c3e656afa7121233f89f52297b573-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal\" (UID: \"c85c3e656afa7121233f89f52297b573\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.758548 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.758504    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:20.836019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.835998    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.839602 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:20.839582    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
May 20 13:50:20.859179 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.859157    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:20.959814 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:20.959787    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:21.060311 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:21.060261    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:21.160789 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:21.160770    2557 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-139-248.ec2.internal\" not found"
May 20 13:50:21.185240 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.185219    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 13:50:21.185704 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.185352    2557 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 13:50:21.251202 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.251178    2557 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:21.264795 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.264779    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 13:50:21.265195 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.265179    2557 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"
May 20 13:50:21.273382 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.273357    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 13:45:20 +0000 UTC" deadline="2027-12-13 22:57:28.590242114 +0000 UTC"
May 20 13:50:21.273382 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.273381    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13737h7m7.316865081s"
May 20 13:50:21.276056 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.276042    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:21.277931 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.277914    2557 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:21.278042 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.277929    2557 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal"
May 20 13:50:21.292474 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.292456    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:21.297683 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.297665    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9wch"
May 20 13:50:21.306744 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.306713    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m9wch"
May 20 13:50:21.373236 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:21.373132    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85c3e656afa7121233f89f52297b573.slice/crio-f3aa5437d6358003194891a71b34133e0b4ab6cf72d78990735d31e830a88ec6 WatchSource:0}: Error finding container f3aa5437d6358003194891a71b34133e0b4ab6cf72d78990735d31e830a88ec6: Status 404 returned error can't find the container with id f3aa5437d6358003194891a71b34133e0b4ab6cf72d78990735d31e830a88ec6
May 20 13:50:21.373407 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:21.373390    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bb2fc9c4d251ff13095ab5d1d57128.slice/crio-d3f37356853303fea8e69b6a5d647ac2730fc1611cc494d6edeeb7ea70237867 WatchSource:0}: Error finding container d3f37356853303fea8e69b6a5d647ac2730fc1611cc494d6edeeb7ea70237867: Status 404 returned error can't find the container with id d3f37356853303fea8e69b6a5d647ac2730fc1611cc494d6edeeb7ea70237867
May 20 13:50:21.376346 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.376333    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:50:21.402975 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.402935    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" event={"ID":"c7bb2fc9c4d251ff13095ab5d1d57128","Type":"ContainerStarted","Data":"d3f37356853303fea8e69b6a5d647ac2730fc1611cc494d6edeeb7ea70237867"}
May 20 13:50:21.403887 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.403866    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" event={"ID":"c85c3e656afa7121233f89f52297b573","Type":"ContainerStarted","Data":"f3aa5437d6358003194891a71b34133e0b4ab6cf72d78990735d31e830a88ec6"}
May 20 13:50:21.645822 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.645765    2557 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:21.743728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:21.743702    2557 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:22.246225 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.246196    2557 apiserver.go:52] "Watching apiserver"
May 20 13:50:22.260435 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.260409    2557 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 13:50:22.262382 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.262344    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-58wjx","openshift-image-registry/node-ca-ww2l9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal","openshift-multus/multus-additional-cni-plugins-th9gl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv","openshift-cluster-node-tuning-operator/tuned-62pdb","openshift-multus/multus-p9tc9","openshift-multus/network-metrics-daemon-p9kp9","openshift-network-diagnostics/network-check-target-wzjlt","openshift-network-operator/iptables-alerter-n8fcf","openshift-ovn-kubernetes/ovnkube-node-gv7zm","kube-system/konnectivity-agent-wc5n2","kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal"]
May 20 13:50:22.263841 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.263820    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.265027 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.265008    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.266177 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.266160    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.267138 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.266966    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 13:50:22.267138 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.266976    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 13:50:22.267138 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.266964    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ldbtc\""
May 20 13:50:22.267138 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.266983    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 13:50:22.267138 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.266988    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 13:50:22.267436 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.267378    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 13:50:22.267489 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.267475    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 13:50:22.267695 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.267679    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.268214 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.268200    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 13:50:22.268294 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.268278    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8w7l9\""
May 20 13:50:22.269057 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.268756    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 13:50:22.269057 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.268895    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 13:50:22.269057 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.268984    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fpxpd\""
May 20 13:50:22.270226 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.269951    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.270226 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.270034    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 13:50:22.270226 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.270077    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.270226 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.270174    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 13:50:22.270500 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.270483    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sg95f\""
May 20 13:50:22.270629 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.270608    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 13:50:22.271329 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.271308    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:22.271419 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.271372    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:22.272758 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.272694    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bfqjm\""
May 20 13:50:22.272758 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.272711    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 13:50:22.272758 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.272727    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7qzbk\""
May 20 13:50:22.272967 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.272843    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 13:50:22.272967 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.272897    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 13:50:22.272967 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.272929    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:22.274087 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.274069    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:22.274179 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.274139    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:22.274179 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.274167    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.275969 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.275950    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.276366 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276344    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-run-k8s-cni-cncf-io\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.276465 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276381    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-var-lib-cni-multus\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.276465 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-system-cni-dir\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.276582 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276461    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-etc-kubernetes\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.276582 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgj9\" (UniqueName: \"kubernetes.io/projected/8be3ff41-5ba1-4283-877d-529de6b51fc7-kube-api-access-ldgj9\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.276582 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-socket-dir-parent\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.276582 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276551    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-run-netns\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.276762 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276597    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-conf-dir\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.276762 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276636    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:22.276762 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276636    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-daemon-config\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.276762 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276678    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8b51a35-4910-4bcf-9198-c36444ad8dd5-cni-binary-copy\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.276971 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.276703    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8be3ff41-5ba1-4283-877d-529de6b51fc7-cni-binary-copy\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.277476 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-run-multus-certs\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.277476 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277125    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-cni-dir\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.277476 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277182    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-system-cni-dir\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.277476 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277219    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-cnibin\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.277476 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-os-release\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.277743 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277483    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-var-lib-cni-bin\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.277743 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277525    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-var-lib-kubelet\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.277743 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277644    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-98fpp\""
May 20 13:50:22.277743 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-cnibin\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.277743 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-hostroot\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.278024 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277722    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-os-release\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.278024 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277834    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.278024 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277872    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 13:50:22.278024 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8b51a35-4910-4bcf-9198-c36444ad8dd5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.278024 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.277992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8b51a35-4910-4bcf-9198-c36444ad8dd5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.278024 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.278022    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/622157d3-3855-4995-8334-d8407eaffade-host\") pod \"node-ca-ww2l9\" (UID: \"622157d3-3855-4995-8334-d8407eaffade\") " pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.278293 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.278123    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/622157d3-3855-4995-8334-d8407eaffade-serviceca\") pod \"node-ca-ww2l9\" (UID: \"622157d3-3855-4995-8334-d8407eaffade\") " pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.278293 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.278158    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dkf5\" (UniqueName: \"kubernetes.io/projected/622157d3-3855-4995-8334-d8407eaffade-kube-api-access-6dkf5\") pod \"node-ca-ww2l9\" (UID: \"622157d3-3855-4995-8334-d8407eaffade\") " pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.278293 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.278187    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxsw2\" (UniqueName: \"kubernetes.io/projected/c8b51a35-4910-4bcf-9198-c36444ad8dd5-kube-api-access-gxsw2\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.278293 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.278269    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:22.278696 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.278474    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 13:50:22.279193 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.278964    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 13:50:22.280701 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.280679    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 13:50:22.280794 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.280723    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 13:50:22.280867 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.280824    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qjs29\""
May 20 13:50:22.281837 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.280692    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 13:50:22.281837 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.281249    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 13:50:22.281837 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.281307    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 13:50:22.282144 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.282097    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 13:50:22.282265 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.282250    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 13:50:22.282343 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.282331    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-grmr7\""
May 20 13:50:22.307371 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.307344    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:21 +0000 UTC" deadline="2027-10-28 16:09:26.485724499 +0000 UTC"
May 20 13:50:22.307371 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.307370    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12626h19m4.178357336s"
May 20 13:50:22.365118 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.365100    2557 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:22.365816 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.365802    2557 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 13:50:22.378477 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-run-systemd\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.378584 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378486    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-etc-openvswitch\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.378584 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378509    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-ovn-node-metrics-cert\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.378584 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378560    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8b51a35-4910-4bcf-9198-c36444ad8dd5-cni-binary-copy\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.378726 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378589    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rpn\" (UniqueName: \"kubernetes.io/projected/1d571823-a33f-4c2b-8524-daa93bb8cb5e-kube-api-access-g6rpn\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.378726 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378611    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d61543-18a2-43a7-b322-54bdaf92f4a9-tmp-dir\") pod \"node-resolver-58wjx\" (UID: \"81d61543-18a2-43a7-b322-54bdaf92f4a9\") " pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.378726 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-sysctl-conf\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.378726 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378658    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-var-lib-openvswitch\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.378726 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378681    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-ovnkube-script-lib\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378732    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-cni-dir\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-sysctl-d\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378795    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-cni-bin\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378821    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-system-cni-dir\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378830    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-cni-dir\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378868    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-os-release\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378896    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-system-cni-dir\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378897    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-var-lib-kubelet\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378934    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-var-lib-kubelet\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.378952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-systemd\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378977    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxlk5\" (UniqueName: \"kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5\") pod \"network-check-target-wzjlt\" (UID: \"36b3619d-2709-46bd-af24-93312d906c20\") " pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.378980    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-os-release\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-systemd-units\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379030    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-hostroot\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379057    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-cni-netd\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379087    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8b51a35-4910-4bcf-9198-c36444ad8dd5-cni-binary-copy\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379084    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-os-release\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379113    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-hostroot\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379117    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379221    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-os-release\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379228    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8b51a35-4910-4bcf-9198-c36444ad8dd5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379268    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379290    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.379351 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-socket-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379392    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-lib-modules\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379435    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/612be1be-2df4-4a7d-a498-87ad24a63762-iptables-alerter-script\") pod \"iptables-alerter-n8fcf\" (UID: \"612be1be-2df4-4a7d-a498-87ad24a63762\") " pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379508    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6dkf5\" (UniqueName: \"kubernetes.io/projected/622157d3-3855-4995-8334-d8407eaffade-kube-api-access-6dkf5\") pod \"node-ca-ww2l9\" (UID: \"622157d3-3855-4995-8334-d8407eaffade\") " pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379533    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-var-lib-cni-multus\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379560    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-run-ovn-kubernetes\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379588    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379606    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-var-lib-cni-multus\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379616    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-system-cni-dir\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379647    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-etc-kubernetes\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379665    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-system-cni-dir\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379673    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgj9\" (UniqueName: \"kubernetes.io/projected/8be3ff41-5ba1-4283-877d-529de6b51fc7-kube-api-access-ldgj9\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379704    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c8b51a35-4910-4bcf-9198-c36444ad8dd5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379715    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-run-openvswitch\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379728    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-etc-kubernetes\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379743    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-run-netns\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379768    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-conf-dir\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379794    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-daemon-config\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379824    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-registration-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-sysconfig\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379869    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-conf-dir\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-kubernetes\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379919    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-run-netns\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379919    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8be3ff41-5ba1-4283-877d-529de6b51fc7-cni-binary-copy\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379957    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-run-multus-certs\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.379985    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-device-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380003    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-run-multus-certs\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-run-ovn\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-env-overrides\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380059    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-modprobe-d\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-tuned\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380123    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr588\" (UniqueName: \"kubernetes.io/projected/438841d1-e772-4acf-b0a6-58e5afe1937f-kube-api-access-pr588\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380149    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-run-netns\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.380811 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-ovnkube-config\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380206    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-cnibin\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380252    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-var-lib-cni-bin\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380276    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81d61543-18a2-43a7-b322-54bdaf92f4a9-hosts-file\") pod \"node-resolver-58wjx\" (UID: \"81d61543-18a2-43a7-b322-54bdaf92f4a9\") " pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380295    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrqk5\" (UniqueName: \"kubernetes.io/projected/03b29b31-e26c-4dfc-ba2f-ca176905526b-kube-api-access-xrqk5\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380327    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-var-lib-cni-bin\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380328    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/612be1be-2df4-4a7d-a498-87ad24a63762-host-slash\") pod \"iptables-alerter-n8fcf\" (UID: \"612be1be-2df4-4a7d-a498-87ad24a63762\") " pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380361    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-cnibin\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380365    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-daemon-config\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380375    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18cd97d3-32d6-4d7a-9c27-5d014aff0dbb-agent-certs\") pod \"konnectivity-agent-wc5n2\" (UID: \"18cd97d3-32d6-4d7a-9c27-5d014aff0dbb\") " pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380397    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8be3ff41-5ba1-4283-877d-529de6b51fc7-cni-binary-copy\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380400    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-cnibin\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380421    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8b51a35-4910-4bcf-9198-c36444ad8dd5-cnibin\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380452    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-sys-fs\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380504    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-run\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380527    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-var-lib-kubelet\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.381518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-log-socket\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380576    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8b51a35-4910-4bcf-9198-c36444ad8dd5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380610    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-sys\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380683    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8p6\" (UniqueName: \"kubernetes.io/projected/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-kube-api-access-zk8p6\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380722    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/622157d3-3855-4995-8334-d8407eaffade-host\") pod \"node-ca-ww2l9\" (UID: \"622157d3-3855-4995-8334-d8407eaffade\") " pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380760    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/622157d3-3855-4995-8334-d8407eaffade-serviceca\") pod \"node-ca-ww2l9\" (UID: \"622157d3-3855-4995-8334-d8407eaffade\") " pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380788    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gxsw2\" (UniqueName: \"kubernetes.io/projected/c8b51a35-4910-4bcf-9198-c36444ad8dd5-kube-api-access-gxsw2\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380827    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/622157d3-3855-4995-8334-d8407eaffade-host\") pod \"node-ca-ww2l9\" (UID: \"622157d3-3855-4995-8334-d8407eaffade\") " pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380878    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-run-k8s-cni-cncf-io\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-host\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380927    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8zr\" (UniqueName: \"kubernetes.io/projected/612be1be-2df4-4a7d-a498-87ad24a63762-kube-api-access-bv8zr\") pod \"iptables-alerter-n8fcf\" (UID: \"612be1be-2df4-4a7d-a498-87ad24a63762\") " pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380950    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-kubelet\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380972    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-slash\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.380994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk2rn\" (UniqueName: \"kubernetes.io/projected/81d61543-18a2-43a7-b322-54bdaf92f4a9-kube-api-access-wk2rn\") pod \"node-resolver-58wjx\" (UID: \"81d61543-18a2-43a7-b322-54bdaf92f4a9\") " pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.381018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-node-log\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.381025    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-host-run-k8s-cni-cncf-io\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.381029    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c8b51a35-4910-4bcf-9198-c36444ad8dd5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.382046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.381061    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18cd97d3-32d6-4d7a-9c27-5d014aff0dbb-konnectivity-ca\") pod \"konnectivity-agent-wc5n2\" (UID: \"18cd97d3-32d6-4d7a-9c27-5d014aff0dbb\") " pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:22.382589 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.381089    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-socket-dir-parent\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.382589 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.381113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03b29b31-e26c-4dfc-ba2f-ca176905526b-tmp\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.382589 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.381152    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8be3ff41-5ba1-4283-877d-529de6b51fc7-multus-socket-dir-parent\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.382589 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.381199    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/622157d3-3855-4995-8334-d8407eaffade-serviceca\") pod \"node-ca-ww2l9\" (UID: \"622157d3-3855-4995-8334-d8407eaffade\") " pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.391018 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.390991    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 13:50:22.394064 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.394018    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgj9\" (UniqueName: \"kubernetes.io/projected/8be3ff41-5ba1-4283-877d-529de6b51fc7-kube-api-access-ldgj9\") pod \"multus-p9tc9\" (UID: \"8be3ff41-5ba1-4283-877d-529de6b51fc7\") " pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.394146 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.394053    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dkf5\" (UniqueName: \"kubernetes.io/projected/622157d3-3855-4995-8334-d8407eaffade-kube-api-access-6dkf5\") pod \"node-ca-ww2l9\" (UID: \"622157d3-3855-4995-8334-d8407eaffade\") " pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.394146 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.394066    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxsw2\" (UniqueName: \"kubernetes.io/projected/c8b51a35-4910-4bcf-9198-c36444ad8dd5-kube-api-access-gxsw2\") pod \"multus-additional-cni-plugins-th9gl\" (UID: \"c8b51a35-4910-4bcf-9198-c36444ad8dd5\") " pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.482234 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482206    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-ovnkube-script-lib\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482384 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482241    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-sysctl-d\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.482384 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482266    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-cni-bin\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482384 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482292    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-systemd\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.482384 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482316    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlk5\" (UniqueName: \"kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5\") pod \"network-check-target-wzjlt\" (UID: \"36b3619d-2709-46bd-af24-93312d906c20\") " pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:22.482384 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482340    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-systemd-units\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482535 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482387    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-cni-netd\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482535 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482390    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-cni-bin\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482535 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482414    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.482535 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482439    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-socket-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.482535 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482461    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-systemd\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.482535 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482463    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-lib-modules\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.482535 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482506    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482535    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/612be1be-2df4-4a7d-a498-87ad24a63762-iptables-alerter-script\") pod \"iptables-alerter-n8fcf\" (UID: \"612be1be-2df4-4a7d-a498-87ad24a63762\") " pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482566    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-run-ovn-kubernetes\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482580    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-lib-modules\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482592    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482625    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-run-openvswitch\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482655    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-registration-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482661    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-sysctl-d\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482681    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-sysconfig\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482690    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482705    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-kubernetes\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482732    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-run-openvswitch\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482746    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-device-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482771    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-run-ovn\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482781    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-socket-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482796    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-env-overrides\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.482803    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:22.482836 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482823    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-modprobe-d\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482830    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-run-ovn-kubernetes\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.482848    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs podName:438841d1-e772-4acf-b0a6-58e5afe1937f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:22.982832108 +0000 UTC m=+3.120386341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs") pod "network-metrics-daemon-p9kp9" (UID: "438841d1-e772-4acf-b0a6-58e5afe1937f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482875    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-tuned\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482889    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-ovnkube-script-lib\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482899    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pr588\" (UniqueName: \"kubernetes.io/projected/438841d1-e772-4acf-b0a6-58e5afe1937f-kube-api-access-pr588\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482917    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-run-netns\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482934    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-ovnkube-config\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482941    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-modprobe-d\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482950    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-sysconfig\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482994    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-run-ovn\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482995    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483042    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-kubernetes\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483043    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-device-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482626    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-systemd-units\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483081    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-registration-dir\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.483300 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482655    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-cni-netd\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.482956    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81d61543-18a2-43a7-b322-54bdaf92f4a9-hosts-file\") pod \"node-resolver-58wjx\" (UID: \"81d61543-18a2-43a7-b322-54bdaf92f4a9\") " pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483131    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xrqk5\" (UniqueName: \"kubernetes.io/projected/03b29b31-e26c-4dfc-ba2f-ca176905526b-kube-api-access-xrqk5\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483157    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/612be1be-2df4-4a7d-a498-87ad24a63762-host-slash\") pod \"iptables-alerter-n8fcf\" (UID: \"612be1be-2df4-4a7d-a498-87ad24a63762\") " pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483182    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18cd97d3-32d6-4d7a-9c27-5d014aff0dbb-agent-certs\") pod \"konnectivity-agent-wc5n2\" (UID: \"18cd97d3-32d6-4d7a-9c27-5d014aff0dbb\") " pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483207    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483233    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-sys-fs\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483259    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-run\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483283    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-var-lib-kubelet\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483309    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-log-socket\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483334    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-sys\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483342    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/612be1be-2df4-4a7d-a498-87ad24a63762-iptables-alerter-script\") pod \"iptables-alerter-n8fcf\" (UID: \"612be1be-2df4-4a7d-a498-87ad24a63762\") " pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483359    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8p6\" (UniqueName: \"kubernetes.io/projected/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-kube-api-access-zk8p6\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483390    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-host\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483414    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8zr\" (UniqueName: \"kubernetes.io/projected/612be1be-2df4-4a7d-a498-87ad24a63762-kube-api-access-bv8zr\") pod \"iptables-alerter-n8fcf\" (UID: \"612be1be-2df4-4a7d-a498-87ad24a63762\") " pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483436    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-env-overrides\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483438    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-kubelet\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484023 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483469    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-kubelet\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483493    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-slash\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483524    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/612be1be-2df4-4a7d-a498-87ad24a63762-host-slash\") pod \"iptables-alerter-n8fcf\" (UID: \"612be1be-2df4-4a7d-a498-87ad24a63762\") " pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483529    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-run\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483547    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wk2rn\" (UniqueName: \"kubernetes.io/projected/81d61543-18a2-43a7-b322-54bdaf92f4a9-kube-api-access-wk2rn\") pod \"node-resolver-58wjx\" (UID: \"81d61543-18a2-43a7-b322-54bdaf92f4a9\") " pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483564    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-var-lib-kubelet\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483576    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-node-log\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483584    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-run-netns\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483602    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18cd97d3-32d6-4d7a-9c27-5d014aff0dbb-konnectivity-ca\") pod \"konnectivity-agent-wc5n2\" (UID: \"18cd97d3-32d6-4d7a-9c27-5d014aff0dbb\") " pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483622    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-host-slash\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483630    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03b29b31-e26c-4dfc-ba2f-ca176905526b-tmp\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483632    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-sys\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483632    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-etc-selinux\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483665    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-run-systemd\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483698    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-etc-openvswitch\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483730    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-log-socket\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483669    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d571823-a33f-4c2b-8524-daa93bb8cb5e-sys-fs\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483776    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-ovn-node-metrics-cert\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.484728 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483789    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-host\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483804    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rpn\" (UniqueName: \"kubernetes.io/projected/1d571823-a33f-4c2b-8524-daa93bb8cb5e-kube-api-access-g6rpn\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483354    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81d61543-18a2-43a7-b322-54bdaf92f4a9-hosts-file\") pod \"node-resolver-58wjx\" (UID: \"81d61543-18a2-43a7-b322-54bdaf92f4a9\") " pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483830    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d61543-18a2-43a7-b322-54bdaf92f4a9-tmp-dir\") pod \"node-resolver-58wjx\" (UID: \"81d61543-18a2-43a7-b322-54bdaf92f4a9\") " pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483837    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-run-systemd\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483889    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-etc-openvswitch\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483909    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-sysctl-conf\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.483948    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-var-lib-openvswitch\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.484011    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-var-lib-openvswitch\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.484115    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-node-log\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.484123    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-sysctl-conf\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.484149    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-ovnkube-config\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.484551    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d61543-18a2-43a7-b322-54bdaf92f4a9-tmp-dir\") pod \"node-resolver-58wjx\" (UID: \"81d61543-18a2-43a7-b322-54bdaf92f4a9\") " pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.485311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.484710    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18cd97d3-32d6-4d7a-9c27-5d014aff0dbb-konnectivity-ca\") pod \"konnectivity-agent-wc5n2\" (UID: \"18cd97d3-32d6-4d7a-9c27-5d014aff0dbb\") " pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:22.485774 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.485751    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03b29b31-e26c-4dfc-ba2f-ca176905526b-etc-tuned\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.486214 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.486185    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03b29b31-e26c-4dfc-ba2f-ca176905526b-tmp\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.486485 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.486439    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18cd97d3-32d6-4d7a-9c27-5d014aff0dbb-agent-certs\") pod \"konnectivity-agent-wc5n2\" (UID: \"18cd97d3-32d6-4d7a-9c27-5d014aff0dbb\") " pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:22.486706 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.486686    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-ovn-node-metrics-cert\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.494695 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.494675    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:22.494695 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.494696    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:22.494956 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.494707    2557 projected.go:194] Error preparing data for projected volume kube-api-access-mxlk5 for pod openshift-network-diagnostics/network-check-target-wzjlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:22.494956 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.494762    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5 podName:36b3619d-2709-46bd-af24-93312d906c20 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:22.994749236 +0000 UTC m=+3.132303432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mxlk5" (UniqueName: "kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5") pod "network-check-target-wzjlt" (UID: "36b3619d-2709-46bd-af24-93312d906c20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:22.500659 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.500314    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8p6\" (UniqueName: \"kubernetes.io/projected/3c2fa842-0cdc-445c-ad8b-bc3050ed727a-kube-api-access-zk8p6\") pod \"ovnkube-node-gv7zm\" (UID: \"3c2fa842-0cdc-445c-ad8b-bc3050ed727a\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.500659 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.500613    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8zr\" (UniqueName: \"kubernetes.io/projected/612be1be-2df4-4a7d-a498-87ad24a63762-kube-api-access-bv8zr\") pod \"iptables-alerter-n8fcf\" (UID: \"612be1be-2df4-4a7d-a498-87ad24a63762\") " pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.501031 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.501008    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk2rn\" (UniqueName: \"kubernetes.io/projected/81d61543-18a2-43a7-b322-54bdaf92f4a9-kube-api-access-wk2rn\") pod \"node-resolver-58wjx\" (UID: \"81d61543-18a2-43a7-b322-54bdaf92f4a9\") " pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.501179 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.501155    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrqk5\" (UniqueName: \"kubernetes.io/projected/03b29b31-e26c-4dfc-ba2f-ca176905526b-kube-api-access-xrqk5\") pod \"tuned-62pdb\" (UID: \"03b29b31-e26c-4dfc-ba2f-ca176905526b\") " pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.501770 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.501746    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr588\" (UniqueName: \"kubernetes.io/projected/438841d1-e772-4acf-b0a6-58e5afe1937f-kube-api-access-pr588\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:22.501888 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.501846    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rpn\" (UniqueName: \"kubernetes.io/projected/1d571823-a33f-4c2b-8524-daa93bb8cb5e-kube-api-access-g6rpn\") pod \"aws-ebs-csi-driver-node-hsmzv\" (UID: \"1d571823-a33f-4c2b-8524-daa93bb8cb5e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.576048 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.576024    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p9tc9"
May 20 13:50:22.583536 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.583514    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ww2l9"
May 20 13:50:22.591078 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.591057    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-th9gl"
May 20 13:50:22.596584 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.596566    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv"
May 20 13:50:22.602117 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.602101    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-62pdb"
May 20 13:50:22.608595 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.608580    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-58wjx"
May 20 13:50:22.615074 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.615058    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n8fcf"
May 20 13:50:22.621604 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.621588    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:22.627166 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.627146    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:22.911108 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:22.911078    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b29b31_e26c_4dfc_ba2f_ca176905526b.slice/crio-9a99d1902c123ff3229cf3dc08b1b278c7ab917ec5c8691f0034f8ad0eb27376 WatchSource:0}: Error finding container 9a99d1902c123ff3229cf3dc08b1b278c7ab917ec5c8691f0034f8ad0eb27376: Status 404 returned error can't find the container with id 9a99d1902c123ff3229cf3dc08b1b278c7ab917ec5c8691f0034f8ad0eb27376
May 20 13:50:22.912466 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:22.912362    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612be1be_2df4_4a7d_a498_87ad24a63762.slice/crio-86717b8c5c841c6d9b141298aaa4f4dc74d1b6b9de61e1904e943603e99abd48 WatchSource:0}: Error finding container 86717b8c5c841c6d9b141298aaa4f4dc74d1b6b9de61e1904e943603e99abd48: Status 404 returned error can't find the container with id 86717b8c5c841c6d9b141298aaa4f4dc74d1b6b9de61e1904e943603e99abd48
May 20 13:50:22.916317 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:22.916290    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be3ff41_5ba1_4283_877d_529de6b51fc7.slice/crio-24aa4479600eea14f635c132c0b4c976de4a2a5beb2f919dcfe2e46949a61ceb WatchSource:0}: Error finding container 24aa4479600eea14f635c132c0b4c976de4a2a5beb2f919dcfe2e46949a61ceb: Status 404 returned error can't find the container with id 24aa4479600eea14f635c132c0b4c976de4a2a5beb2f919dcfe2e46949a61ceb
May 20 13:50:22.916996 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:22.916959    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d571823_a33f_4c2b_8524_daa93bb8cb5e.slice/crio-0360f3eb70babd30dff9686124e8a0c852b8cad0b160db1a5a2a4f8f92bd1f61 WatchSource:0}: Error finding container 0360f3eb70babd30dff9686124e8a0c852b8cad0b160db1a5a2a4f8f92bd1f61: Status 404 returned error can't find the container with id 0360f3eb70babd30dff9686124e8a0c852b8cad0b160db1a5a2a4f8f92bd1f61
May 20 13:50:22.917940 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:22.917916    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cd97d3_32d6_4d7a_9c27_5d014aff0dbb.slice/crio-7ac993ef8e3bb05fb774aa6c653f3e1cada701688bd68ee4a1c57a655bfb553a WatchSource:0}: Error finding container 7ac993ef8e3bb05fb774aa6c653f3e1cada701688bd68ee4a1c57a655bfb553a: Status 404 returned error can't find the container with id 7ac993ef8e3bb05fb774aa6c653f3e1cada701688bd68ee4a1c57a655bfb553a
May 20 13:50:22.918646 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:22.918625    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622157d3_3855_4995_8334_d8407eaffade.slice/crio-f788b0fc657e3975dd23ae9a24de1af7aa1467ebabcc38dfec4e6e0a44f83c13 WatchSource:0}: Error finding container f788b0fc657e3975dd23ae9a24de1af7aa1467ebabcc38dfec4e6e0a44f83c13: Status 404 returned error can't find the container with id f788b0fc657e3975dd23ae9a24de1af7aa1467ebabcc38dfec4e6e0a44f83c13
May 20 13:50:22.919490 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:22.919470    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2fa842_0cdc_445c_ad8b_bc3050ed727a.slice/crio-5240a5bfd71d3f19ed40faa6b54e697d19ca97adf814c3145acdd9064a26d9f1 WatchSource:0}: Error finding container 5240a5bfd71d3f19ed40faa6b54e697d19ca97adf814c3145acdd9064a26d9f1: Status 404 returned error can't find the container with id 5240a5bfd71d3f19ed40faa6b54e697d19ca97adf814c3145acdd9064a26d9f1
May 20 13:50:22.920475 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:22.920442    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d61543_18a2_43a7_b322_54bdaf92f4a9.slice/crio-4fd2f503650b7fec573f34cacf35119c5c906444741efc596676eff1268bac36 WatchSource:0}: Error finding container 4fd2f503650b7fec573f34cacf35119c5c906444741efc596676eff1268bac36: Status 404 returned error can't find the container with id 4fd2f503650b7fec573f34cacf35119c5c906444741efc596676eff1268bac36
May 20 13:50:22.921441 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:22.921403    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b51a35_4910_4bcf_9198_c36444ad8dd5.slice/crio-d027abb3acd892edd00e1f14a5071f3403570e200effd58912c6cb361eb5c8b4 WatchSource:0}: Error finding container d027abb3acd892edd00e1f14a5071f3403570e200effd58912c6cb361eb5c8b4: Status 404 returned error can't find the container with id d027abb3acd892edd00e1f14a5071f3403570e200effd58912c6cb361eb5c8b4
May 20 13:50:22.987384 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:22.987365    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:22.987462 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.987456    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:22.987509 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:22.987500    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs podName:438841d1-e772-4acf-b0a6-58e5afe1937f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:23.98748453 +0000 UTC m=+4.125038726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs") pod "network-metrics-daemon-p9kp9" (UID: "438841d1-e772-4acf-b0a6-58e5afe1937f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:23.088611 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.088584    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlk5\" (UniqueName: \"kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5\") pod \"network-check-target-wzjlt\" (UID: \"36b3619d-2709-46bd-af24-93312d906c20\") " pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:23.088712 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:23.088698    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:23.088771 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:23.088718    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:23.088771 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:23.088730    2557 projected.go:194] Error preparing data for projected volume kube-api-access-mxlk5 for pod openshift-network-diagnostics/network-check-target-wzjlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:23.088838 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:23.088773    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5 podName:36b3619d-2709-46bd-af24-93312d906c20 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:24.088760184 +0000 UTC m=+4.226314382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxlk5" (UniqueName: "kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5") pod "network-check-target-wzjlt" (UID: "36b3619d-2709-46bd-af24-93312d906c20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:23.308364 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.308231    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:21 +0000 UTC" deadline="2028-02-27 01:05:18.821872838 +0000 UTC"
May 20 13:50:23.308364 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.308287    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15539h14m55.513590805s"
May 20 13:50:23.400792 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.400730    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:23.400947 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:23.400846    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:23.416339 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.416308    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" event={"ID":"c7bb2fc9c4d251ff13095ab5d1d57128","Type":"ContainerStarted","Data":"b4ed7b534056a13dced1a2fbd4925858b5f24941869f27662d5b85bfbb792f54"}
May 20 13:50:23.440278 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.440239    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-th9gl" event={"ID":"c8b51a35-4910-4bcf-9198-c36444ad8dd5","Type":"ContainerStarted","Data":"d027abb3acd892edd00e1f14a5071f3403570e200effd58912c6cb361eb5c8b4"}
May 20 13:50:23.448987 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.448954    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-58wjx" event={"ID":"81d61543-18a2-43a7-b322-54bdaf92f4a9","Type":"ContainerStarted","Data":"4fd2f503650b7fec573f34cacf35119c5c906444741efc596676eff1268bac36"}
May 20 13:50:23.453266 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.453205    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerStarted","Data":"5240a5bfd71d3f19ed40faa6b54e697d19ca97adf814c3145acdd9064a26d9f1"}
May 20 13:50:23.461653 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.461609    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wc5n2" event={"ID":"18cd97d3-32d6-4d7a-9c27-5d014aff0dbb","Type":"ContainerStarted","Data":"7ac993ef8e3bb05fb774aa6c653f3e1cada701688bd68ee4a1c57a655bfb553a"}
May 20 13:50:23.468002 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.467898    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p9tc9" event={"ID":"8be3ff41-5ba1-4283-877d-529de6b51fc7","Type":"ContainerStarted","Data":"24aa4479600eea14f635c132c0b4c976de4a2a5beb2f919dcfe2e46949a61ceb"}
May 20 13:50:23.469785 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.469747    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n8fcf" event={"ID":"612be1be-2df4-4a7d-a498-87ad24a63762","Type":"ContainerStarted","Data":"86717b8c5c841c6d9b141298aaa4f4dc74d1b6b9de61e1904e943603e99abd48"}
May 20 13:50:23.473163 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.473138    2557 generic.go:358] "Generic (PLEG): container finished" podID="c85c3e656afa7121233f89f52297b573" containerID="3fd98ce5cd88773401c5049a2e776b0ea275d08863c6cb03741ed12a9073ed64" exitCode=0
May 20 13:50:23.473673 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.473653    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" event={"ID":"c85c3e656afa7121233f89f52297b573","Type":"ContainerDied","Data":"3fd98ce5cd88773401c5049a2e776b0ea275d08863c6cb03741ed12a9073ed64"}
May 20 13:50:23.481099 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.481074    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ww2l9" event={"ID":"622157d3-3855-4995-8334-d8407eaffade","Type":"ContainerStarted","Data":"f788b0fc657e3975dd23ae9a24de1af7aa1467ebabcc38dfec4e6e0a44f83c13"}
May 20 13:50:23.496415 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.496370    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-248.ec2.internal" podStartSLOduration=2.496354213 podStartE2EDuration="2.496354213s" podCreationTimestamp="2026-05-20 13:50:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:23.436711905 +0000 UTC m=+3.574266128" watchObservedRunningTime="2026-05-20 13:50:23.496354213 +0000 UTC m=+3.633908431"
May 20 13:50:23.498233 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.498208    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv" event={"ID":"1d571823-a33f-4c2b-8524-daa93bb8cb5e","Type":"ContainerStarted","Data":"0360f3eb70babd30dff9686124e8a0c852b8cad0b160db1a5a2a4f8f92bd1f61"}
May 20 13:50:23.518381 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.518325    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-62pdb" event={"ID":"03b29b31-e26c-4dfc-ba2f-ca176905526b","Type":"ContainerStarted","Data":"9a99d1902c123ff3229cf3dc08b1b278c7ab917ec5c8691f0034f8ad0eb27376"}
May 20 13:50:23.994009 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:23.993383    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:23.994009 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:23.993575    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:23.994009 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:23.993635    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs podName:438841d1-e772-4acf-b0a6-58e5afe1937f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:25.993616806 +0000 UTC m=+6.131171016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs") pod "network-metrics-daemon-p9kp9" (UID: "438841d1-e772-4acf-b0a6-58e5afe1937f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:24.094611 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:24.094581    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlk5\" (UniqueName: \"kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5\") pod \"network-check-target-wzjlt\" (UID: \"36b3619d-2709-46bd-af24-93312d906c20\") " pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:24.094780 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:24.094734    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:24.094780 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:24.094753    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:24.094780 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:24.094763    2557 projected.go:194] Error preparing data for projected volume kube-api-access-mxlk5 for pod openshift-network-diagnostics/network-check-target-wzjlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:24.094960 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:24.094814    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5 podName:36b3619d-2709-46bd-af24-93312d906c20 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:26.094796753 +0000 UTC m=+6.232350949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxlk5" (UniqueName: "kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5") pod "network-check-target-wzjlt" (UID: "36b3619d-2709-46bd-af24-93312d906c20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:24.403059 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:24.401751    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:24.403059 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:24.401900    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:24.539008 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:24.538902    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" event={"ID":"c85c3e656afa7121233f89f52297b573","Type":"ContainerStarted","Data":"bde9c69e1b4feeb57770ee7a939d29749063d049f1cc7a37ea02429691017950"}
May 20 13:50:25.400988 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:25.400930    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:25.401152 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:25.401054    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:26.012293 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:26.012253    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:26.012777 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:26.012396    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:26.012777 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:26.012460    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs podName:438841d1-e772-4acf-b0a6-58e5afe1937f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:30.012442027 +0000 UTC m=+10.149996228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs") pod "network-metrics-daemon-p9kp9" (UID: "438841d1-e772-4acf-b0a6-58e5afe1937f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:26.113484 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:26.113340    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlk5\" (UniqueName: \"kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5\") pod \"network-check-target-wzjlt\" (UID: \"36b3619d-2709-46bd-af24-93312d906c20\") " pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:26.113626 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:26.113514    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:26.113626 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:26.113570    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:26.113626 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:26.113585    2557 projected.go:194] Error preparing data for projected volume kube-api-access-mxlk5 for pod openshift-network-diagnostics/network-check-target-wzjlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:26.113794 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:26.113662    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5 podName:36b3619d-2709-46bd-af24-93312d906c20 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:30.113643004 +0000 UTC m=+10.251197217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxlk5" (UniqueName: "kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5") pod "network-check-target-wzjlt" (UID: "36b3619d-2709-46bd-af24-93312d906c20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:26.401516 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:26.401486    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:26.401690 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:26.401616    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:27.401804 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:27.401073    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:27.401804 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:27.401197    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:28.401202 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:28.401082    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:28.401386 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:28.401219    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:29.401197 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:29.401168    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:29.401651 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:29.401285    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:30.043244 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:30.043195    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:30.043420 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:30.043366    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:30.043484 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:30.043428    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs podName:438841d1-e772-4acf-b0a6-58e5afe1937f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:38.043409286 +0000 UTC m=+18.180963502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs") pod "network-metrics-daemon-p9kp9" (UID: "438841d1-e772-4acf-b0a6-58e5afe1937f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:30.143558 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:30.143494    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlk5\" (UniqueName: \"kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5\") pod \"network-check-target-wzjlt\" (UID: \"36b3619d-2709-46bd-af24-93312d906c20\") " pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:30.143721 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:30.143657    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:30.143721 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:30.143678    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:30.143721 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:30.143690    2557 projected.go:194] Error preparing data for projected volume kube-api-access-mxlk5 for pod openshift-network-diagnostics/network-check-target-wzjlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:30.143894 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:30.143741    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5 podName:36b3619d-2709-46bd-af24-93312d906c20 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:38.143723899 +0000 UTC m=+18.281278101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxlk5" (UniqueName: "kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5") pod "network-check-target-wzjlt" (UID: "36b3619d-2709-46bd-af24-93312d906c20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:30.401522 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:30.401461    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:30.401959 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:30.401591    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:31.400566 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:31.400536    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:31.400744 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:31.400651    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:32.401088 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:32.400804    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:32.401088 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:32.400951    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:33.400674 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:33.400424    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:33.400837 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:33.400777    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:34.401129 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:34.401098    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:34.401557 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:34.401222    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:35.401136 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:35.401104    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:35.401540 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:35.401210    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:36.401483 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:36.401453    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:36.401887 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:36.401556    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:37.401088 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:37.401059    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:37.401314 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:37.401155    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:38.106904 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:38.106875    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:38.107273 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:38.106998    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:38.107273 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:38.107056    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs podName:438841d1-e772-4acf-b0a6-58e5afe1937f nodeName:}" failed. No retries permitted until 2026-05-20 13:50:54.107036497 +0000 UTC m=+34.244590695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs") pod "network-metrics-daemon-p9kp9" (UID: "438841d1-e772-4acf-b0a6-58e5afe1937f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:38.207272 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:38.207237    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlk5\" (UniqueName: \"kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5\") pod \"network-check-target-wzjlt\" (UID: \"36b3619d-2709-46bd-af24-93312d906c20\") " pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:38.207421 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:38.207364    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:38.207421 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:38.207383    2557 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:38.207421 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:38.207394    2557 projected.go:194] Error preparing data for projected volume kube-api-access-mxlk5 for pod openshift-network-diagnostics/network-check-target-wzjlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:38.207558 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:38.207451    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5 podName:36b3619d-2709-46bd-af24-93312d906c20 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:54.207434188 +0000 UTC m=+34.344988391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mxlk5" (UniqueName: "kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5") pod "network-check-target-wzjlt" (UID: "36b3619d-2709-46bd-af24-93312d906c20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:38.400905 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:38.400816    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:38.401058 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:38.400958    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:39.401312 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:39.401282    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:39.401727 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:39.401402    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:40.402415 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:40.402023    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:40.402415 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:40.402144    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:40.572156 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:40.571940    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wc5n2" event={"ID":"18cd97d3-32d6-4d7a-9c27-5d014aff0dbb","Type":"ContainerStarted","Data":"cfb2ad89a336b1f4d896970086ec4a6c4e90da06dbec9aebe3a69c6480c45fdf"}
May 20 13:50:40.573648 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:40.573623    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ww2l9" event={"ID":"622157d3-3855-4995-8334-d8407eaffade","Type":"ContainerStarted","Data":"c95150c633d2df962b1b7196999f690e79554ebef74d1fffbe7d2fcb00fe0165"}
May 20 13:50:40.574919 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:40.574887    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv" event={"ID":"1d571823-a33f-4c2b-8524-daa93bb8cb5e","Type":"ContainerStarted","Data":"914146ad87f313c307752f7bbec96e71e8d78142a5674503431e30924d41df90"}
May 20 13:50:40.577546 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:40.577197    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-62pdb" event={"ID":"03b29b31-e26c-4dfc-ba2f-ca176905526b","Type":"ContainerStarted","Data":"7eca6c8aa1cb0551e3be3f649892e7d803b19a9c0d1211a0e2cb7f47c8acb7f0"}
May 20 13:50:40.588139 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:40.588105    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-248.ec2.internal" podStartSLOduration=19.5880952 podStartE2EDuration="19.5880952s" podCreationTimestamp="2026-05-20 13:50:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:24.555383359 +0000 UTC m=+4.692937578" watchObservedRunningTime="2026-05-20 13:50:40.5880952 +0000 UTC m=+20.725649417"
May 20 13:50:40.588641 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:40.588614    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wc5n2" podStartSLOduration=3.211493902 podStartE2EDuration="20.58860514s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:22.920100703 +0000 UTC m=+3.057654900" lastFinishedPulling="2026-05-20 13:50:40.297211927 +0000 UTC m=+20.434766138" observedRunningTime="2026-05-20 13:50:40.58801235 +0000 UTC m=+20.725566568" watchObservedRunningTime="2026-05-20 13:50:40.58860514 +0000 UTC m=+20.726159359"
May 20 13:50:40.606461 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:40.606417    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-62pdb" podStartSLOduration=3.183559526 podStartE2EDuration="20.606404251s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:22.91336432 +0000 UTC m=+3.050918527" lastFinishedPulling="2026-05-20 13:50:40.336209041 +0000 UTC m=+20.473763252" observedRunningTime="2026-05-20 13:50:40.605990009 +0000 UTC m=+20.743544241" watchObservedRunningTime="2026-05-20 13:50:40.606404251 +0000 UTC m=+20.743958470"
May 20 13:50:40.620887 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:40.620835    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ww2l9" podStartSLOduration=3.243888838 podStartE2EDuration="20.620825269s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:22.92037551 +0000 UTC m=+3.057929715" lastFinishedPulling="2026-05-20 13:50:40.297311939 +0000 UTC m=+20.434866146" observedRunningTime="2026-05-20 13:50:40.620376181 +0000 UTC m=+20.757930399" watchObservedRunningTime="2026-05-20 13:50:40.620825269 +0000 UTC m=+20.758379486"
May 20 13:50:41.401132 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.401106    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:41.401286 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:41.401202    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:41.580521 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.580498    2557 generic.go:358] "Generic (PLEG): container finished" podID="c8b51a35-4910-4bcf-9198-c36444ad8dd5" containerID="a53e9a6fb0cb6a138674dd4c57323d131d66e3dda1048195e8ddb18ed87d175f" exitCode=0
May 20 13:50:41.581102 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.580562    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-th9gl" event={"ID":"c8b51a35-4910-4bcf-9198-c36444ad8dd5","Type":"ContainerDied","Data":"a53e9a6fb0cb6a138674dd4c57323d131d66e3dda1048195e8ddb18ed87d175f"}
May 20 13:50:41.582128 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.582097    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-58wjx" event={"ID":"81d61543-18a2-43a7-b322-54bdaf92f4a9","Type":"ContainerStarted","Data":"4c290c476aa06d91cb53e12fc188e587a87cf21e632057531e50fc78ce9a909b"}
May 20 13:50:41.584806 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.584782    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 13:50:41.586060 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.585352    2557 generic.go:358] "Generic (PLEG): container finished" podID="3c2fa842-0cdc-445c-ad8b-bc3050ed727a" containerID="c85062f5c37331a55403bb1473858aa989d593020441fa7aa372bf246f143fa8" exitCode=1
May 20 13:50:41.586060 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.585665    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerStarted","Data":"04436e979f78c25382eb8fb6707fe727205e8ee431d088a6d0e2dbec1de219c9"}
May 20 13:50:41.586060 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.585688    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerStarted","Data":"237dc42700bf29ae40c3d7c12235612740ce4012939669c7be67ad71d1448bc8"}
May 20 13:50:41.586060 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.585702    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerStarted","Data":"b9bafbf71ed223820f6bd117c352c4a244793ea73063f7a55515f0a749dfceff"}
May 20 13:50:41.586060 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.585753    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerStarted","Data":"807c5157ecd1ed0c3928bc6cee01d75106f27d39137f28c6649f68a214fa21e7"}
May 20 13:50:41.586060 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.585768    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerDied","Data":"c85062f5c37331a55403bb1473858aa989d593020441fa7aa372bf246f143fa8"}
May 20 13:50:41.586060 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.585791    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerStarted","Data":"fc70236fdffde3b7591c391d2a8b9f688d2fc5c25c671d4775a99b56e9ee7704"}
May 20 13:50:41.589499 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.589424    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p9tc9" event={"ID":"8be3ff41-5ba1-4283-877d-529de6b51fc7","Type":"ContainerStarted","Data":"c0ab3963b8a3ea52231953da04f7ba4a4eaea62dc4d39872a1014dfa16300998"}
May 20 13:50:41.629279 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.629235    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-p9tc9" podStartSLOduration=3.919681918 podStartE2EDuration="21.629222043s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:22.917946779 +0000 UTC m=+3.055500985" lastFinishedPulling="2026-05-20 13:50:40.627486914 +0000 UTC m=+20.765041110" observedRunningTime="2026-05-20 13:50:41.628965145 +0000 UTC m=+21.766519360" watchObservedRunningTime="2026-05-20 13:50:41.629222043 +0000 UTC m=+21.766776263"
May 20 13:50:41.638759 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:41.638741    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 13:50:42.340046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:42.339937    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T13:50:41.638754365Z","UUID":"48fd308d-b7fa-48ee-b5fc-1e2adea25663","Handler":null,"Name":"","Endpoint":""}
May 20 13:50:42.341642 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:42.341621    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 13:50:42.341826 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:42.341724    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 13:50:42.401506 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:42.401468    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:42.401648 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:42.401600    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:42.592747 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:42.592676    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n8fcf" event={"ID":"612be1be-2df4-4a7d-a498-87ad24a63762","Type":"ContainerStarted","Data":"542f30ce94a4a97c28cca699c7d3b579f2e38a3dff26c98be9b52694279d0a61"}
May 20 13:50:42.594469 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:42.594441    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv" event={"ID":"1d571823-a33f-4c2b-8524-daa93bb8cb5e","Type":"ContainerStarted","Data":"bce727a3f5478f2e872ce30a2db6af0253094feeb0d758e102ec67354e3f4e0a"}
May 20 13:50:42.608518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:42.608482    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n8fcf" podStartSLOduration=5.179230544 podStartE2EDuration="22.608471321s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:22.914030243 +0000 UTC m=+3.051584442" lastFinishedPulling="2026-05-20 13:50:40.343271013 +0000 UTC m=+20.480825219" observedRunningTime="2026-05-20 13:50:42.608249857 +0000 UTC m=+22.745804100" watchObservedRunningTime="2026-05-20 13:50:42.608471321 +0000 UTC m=+22.746025539"
May 20 13:50:42.608719 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:42.608697    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-58wjx" podStartSLOduration=5.191846891 podStartE2EDuration="22.608692208s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:22.922504994 +0000 UTC m=+3.060059206" lastFinishedPulling="2026-05-20 13:50:40.339350323 +0000 UTC m=+20.476904523" observedRunningTime="2026-05-20 13:50:41.644463916 +0000 UTC m=+21.782018134" watchObservedRunningTime="2026-05-20 13:50:42.608692208 +0000 UTC m=+22.746246481"
May 20 13:50:43.400991 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:43.400964    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:43.401199 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:43.401057    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:43.598322 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:43.598274    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv" event={"ID":"1d571823-a33f-4c2b-8524-daa93bb8cb5e","Type":"ContainerStarted","Data":"ebe04027808ca434ab682c8f93513496197d7040df37ad452676e1dbd33cd79b"}
May 20 13:50:43.601314 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:43.601290    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 13:50:43.601714 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:43.601684    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerStarted","Data":"1c92936d8fe0aface84065af81fcdc41f1a8ab7eb24c5f4cb6d61858a66beb76"}
May 20 13:50:43.616984 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:43.616945    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hsmzv" podStartSLOduration=3.940340252 podStartE2EDuration="23.616934623s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:22.918912766 +0000 UTC m=+3.056466979" lastFinishedPulling="2026-05-20 13:50:42.595507141 +0000 UTC m=+22.733061350" observedRunningTime="2026-05-20 13:50:43.616718463 +0000 UTC m=+23.754272681" watchObservedRunningTime="2026-05-20 13:50:43.616934623 +0000 UTC m=+23.754488840"
May 20 13:50:44.135893 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:44.135848    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:44.136715 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:44.136693    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:44.401382 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:44.401245    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:44.401382 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:44.401365    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:45.400750 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:45.400638    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:45.401235 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:45.400843    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:45.606200 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:45.606130    2557 generic.go:358] "Generic (PLEG): container finished" podID="c8b51a35-4910-4bcf-9198-c36444ad8dd5" containerID="f0ad26de9964f7348dd0eb92f125a2c498fd27eb4bfccad5238282299468ba9e" exitCode=0
May 20 13:50:45.606312 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:45.606217    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-th9gl" event={"ID":"c8b51a35-4910-4bcf-9198-c36444ad8dd5","Type":"ContainerDied","Data":"f0ad26de9964f7348dd0eb92f125a2c498fd27eb4bfccad5238282299468ba9e"}
May 20 13:50:45.609328 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:45.609312    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 13:50:45.609630 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:45.609613    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerStarted","Data":"f5d4a608577446b25ec2d284a045d577fbe15736cdcdfaf8dd6baf4c5f887b91"}
May 20 13:50:45.609899 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:45.609881    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:45.610081 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:45.610067    2557 scope.go:117] "RemoveContainer" containerID="c85062f5c37331a55403bb1473858aa989d593020441fa7aa372bf246f143fa8"
May 20 13:50:45.625433 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:45.625416    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:46.400704 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:46.400679    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:46.400818 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:46.400779    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:46.614065 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:46.614020    2557 generic.go:358] "Generic (PLEG): container finished" podID="c8b51a35-4910-4bcf-9198-c36444ad8dd5" containerID="e53e46f6db023e667cd708f01c0f042adb75e92f67448527861909b47cc27f53" exitCode=0
May 20 13:50:46.614409 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:46.614119    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-th9gl" event={"ID":"c8b51a35-4910-4bcf-9198-c36444ad8dd5","Type":"ContainerDied","Data":"e53e46f6db023e667cd708f01c0f042adb75e92f67448527861909b47cc27f53"}
May 20 13:50:46.618294 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:46.618275    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 13:50:46.618712 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:46.618682    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" event={"ID":"3c2fa842-0cdc-445c-ad8b-bc3050ed727a","Type":"ContainerStarted","Data":"b0545ced0df88d6c51a525beea8220963cee77563933c66a76c119268204c6c1"}
May 20 13:50:46.618966 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:46.618947    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:46.619038 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:46.618980    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:46.635001 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:46.634976    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:50:47.342868 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:47.342636    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm" podStartSLOduration=9.895463751 podStartE2EDuration="27.342618779s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:22.922515045 +0000 UTC m=+3.060069246" lastFinishedPulling="2026-05-20 13:50:40.369670065 +0000 UTC m=+20.507224274" observedRunningTime="2026-05-20 13:50:46.670258762 +0000 UTC m=+26.807812996" watchObservedRunningTime="2026-05-20 13:50:47.342618779 +0000 UTC m=+27.480172998"
May 20 13:50:47.343655 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:47.343629    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wzjlt"]
May 20 13:50:47.343793 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:47.343777    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:47.343925 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:47.343903    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:47.347335 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:47.347283    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9kp9"]
May 20 13:50:47.347434 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:47.347393    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:47.347508 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:47.347489    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:47.622985 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:47.622915    2557 generic.go:358] "Generic (PLEG): container finished" podID="c8b51a35-4910-4bcf-9198-c36444ad8dd5" containerID="e02a49a8a43332f2f51dd358e83cd58d36ef76afdd5f009d44e54b69b8419d74" exitCode=0
May 20 13:50:47.623349 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:47.623001    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-th9gl" event={"ID":"c8b51a35-4910-4bcf-9198-c36444ad8dd5","Type":"ContainerDied","Data":"e02a49a8a43332f2f51dd358e83cd58d36ef76afdd5f009d44e54b69b8419d74"}
May 20 13:50:48.401370 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:48.401338    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:48.401529 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:48.401442    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:49.400700 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:49.400669    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:49.401244 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:49.400774    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:50.401931 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:50.401903    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:50.402437 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:50.402017    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:51.386669 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:51.386641    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:51.386823 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:51.386773    2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 20 13:50:51.387299 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:51.387277    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wc5n2"
May 20 13:50:51.400975 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:51.400954    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:51.401057 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:51.401043    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wzjlt" podUID="36b3619d-2709-46bd-af24-93312d906c20"
May 20 13:50:52.401099 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:52.401068    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:52.401686 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:52.401183    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9kp9" podUID="438841d1-e772-4acf-b0a6-58e5afe1937f"
May 20 13:50:53.202046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.202016    2557 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-139-248.ec2.internal" event="NodeReady"
May 20 13:50:53.202144 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.202116    2557 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 13:50:53.250480 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.250460    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8vcdb"]
May 20 13:50:53.272114 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.271927    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4qp54"]
May 20 13:50:53.272226 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.272141    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.275004 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.274981    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 13:50:53.275104 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.275020    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wmcgw\""
May 20 13:50:53.275104 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.275075    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 13:50:53.303348 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.303304    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8vcdb"]
May 20 13:50:53.303348 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.303332    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4qp54"]
May 20 13:50:53.303450 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.303439    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:50:53.306181 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.306162    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 13:50:53.306270 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.306162    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 13:50:53.306270 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.306166    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8tpd8\""
May 20 13:50:53.306407 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.306387    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 13:50:53.401105 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.400939    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:53.404217 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.404193    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hvv45\""
May 20 13:50:53.404354 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.404193    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 13:50:53.404354 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.404193    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 13:50:53.431915 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.431864    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.432049 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.431944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-config-volume\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.432049 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.431962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjqz\" (UniqueName: \"kubernetes.io/projected/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-kube-api-access-lfjqz\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.432049 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.431983    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjm8\" (UniqueName: \"kubernetes.io/projected/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-kube-api-access-lpjm8\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:50:53.432213 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.432063    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-tmp-dir\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.432213 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.432112    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:50:53.533163 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.533140    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:50:53.533260 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.533173    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.533260 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.533200    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-config-volume\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.533260 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.533227    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjqz\" (UniqueName: \"kubernetes.io/projected/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-kube-api-access-lfjqz\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.533260 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.533254    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjm8\" (UniqueName: \"kubernetes.io/projected/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-kube-api-access-lpjm8\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:50:53.533401 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:53.533297    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:53.533401 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.533313    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-tmp-dir\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.533401 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:53.533357    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls podName:ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:54.033339066 +0000 UTC m=+34.170893274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls") pod "dns-default-8vcdb" (UID: "ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75") : secret "dns-default-metrics-tls" not found
May 20 13:50:53.533401 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:53.533296    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:53.533582 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:53.533450    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert podName:0cfdbcdd-0be7-42e1-91eb-9892ee0ea925 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:54.033432672 +0000 UTC m=+34.170986874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert") pod "ingress-canary-4qp54" (UID: "0cfdbcdd-0be7-42e1-91eb-9892ee0ea925") : secret "canary-serving-cert" not found
May 20 13:50:53.533677 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.533655    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-tmp-dir\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.533743 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.533725    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-config-volume\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.545600 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.545582    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjqz\" (UniqueName: \"kubernetes.io/projected/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-kube-api-access-lfjqz\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:53.545882 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.545844    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjm8\" (UniqueName: \"kubernetes.io/projected/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-kube-api-access-lpjm8\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:50:53.636706 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.636677    2557 generic.go:358] "Generic (PLEG): container finished" podID="c8b51a35-4910-4bcf-9198-c36444ad8dd5" containerID="c1a3719078a8ad493a2c17bf04a21786bf970e11987baad103343893541d5d59" exitCode=0
May 20 13:50:53.636794 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:53.636731    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-th9gl" event={"ID":"c8b51a35-4910-4bcf-9198-c36444ad8dd5","Type":"ContainerDied","Data":"c1a3719078a8ad493a2c17bf04a21786bf970e11987baad103343893541d5d59"}
May 20 13:50:54.036563 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.036504    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:50:54.036563 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.036543    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:54.036727 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:54.036627    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:54.036727 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:54.036648    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:54.036727 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:54.036672    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls podName:ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:55.036657566 +0000 UTC m=+35.174211762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls") pod "dns-default-8vcdb" (UID: "ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75") : secret "dns-default-metrics-tls" not found
May 20 13:50:54.036727 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:54.036704    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert podName:0cfdbcdd-0be7-42e1-91eb-9892ee0ea925 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:55.036686752 +0000 UTC m=+35.174240949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert") pod "ingress-canary-4qp54" (UID: "0cfdbcdd-0be7-42e1-91eb-9892ee0ea925") : secret "canary-serving-cert" not found
May 20 13:50:54.136963 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.136935    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:54.137059 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:54.137042    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:54.137093 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:54.137084    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs podName:438841d1-e772-4acf-b0a6-58e5afe1937f nodeName:}" failed. No retries permitted until 2026-05-20 13:51:26.137073993 +0000 UTC m=+66.274628190 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs") pod "network-metrics-daemon-p9kp9" (UID: "438841d1-e772-4acf-b0a6-58e5afe1937f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:54.237960 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.237933    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlk5\" (UniqueName: \"kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5\") pod \"network-check-target-wzjlt\" (UID: \"36b3619d-2709-46bd-af24-93312d906c20\") " pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:54.240331 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.240310    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxlk5\" (UniqueName: \"kubernetes.io/projected/36b3619d-2709-46bd-af24-93312d906c20-kube-api-access-mxlk5\") pod \"network-check-target-wzjlt\" (UID: \"36b3619d-2709-46bd-af24-93312d906c20\") " pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:54.309399 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.309330    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:54.401099 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.401075    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:50:54.404374 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.404318    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:50:54.404925 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.404882    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d6bxt\""
May 20 13:50:54.509029 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.508922    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wzjlt"]
May 20 13:50:54.513097 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:50:54.513060    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b3619d_2709_46bd_af24_93312d906c20.slice/crio-cf9a436ea067ba4c650ee5c78dff8101f9316d6d62887705e480f547f7d91ed1 WatchSource:0}: Error finding container cf9a436ea067ba4c650ee5c78dff8101f9316d6d62887705e480f547f7d91ed1: Status 404 returned error can't find the container with id cf9a436ea067ba4c650ee5c78dff8101f9316d6d62887705e480f547f7d91ed1
May 20 13:50:54.641849 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.641819    2557 generic.go:358] "Generic (PLEG): container finished" podID="c8b51a35-4910-4bcf-9198-c36444ad8dd5" containerID="8fb848bcf36f539b0b827c94c5ce10be2da83cbc446bff5b84d42fa9abfd7691" exitCode=0
May 20 13:50:54.641967 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.641896    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-th9gl" event={"ID":"c8b51a35-4910-4bcf-9198-c36444ad8dd5","Type":"ContainerDied","Data":"8fb848bcf36f539b0b827c94c5ce10be2da83cbc446bff5b84d42fa9abfd7691"}
May 20 13:50:54.643082 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:54.643058    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wzjlt" event={"ID":"36b3619d-2709-46bd-af24-93312d906c20","Type":"ContainerStarted","Data":"cf9a436ea067ba4c650ee5c78dff8101f9316d6d62887705e480f547f7d91ed1"}
May 20 13:50:55.044438 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:55.044380    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:50:55.044438 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:55.044417    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:55.044594 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:55.044524    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:55.044594 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:55.044536    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:55.044669 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:55.044599    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls podName:ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:57.044568118 +0000 UTC m=+37.182122317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls") pod "dns-default-8vcdb" (UID: "ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75") : secret "dns-default-metrics-tls" not found
May 20 13:50:55.044797 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:55.044783    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert podName:0cfdbcdd-0be7-42e1-91eb-9892ee0ea925 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:57.044767685 +0000 UTC m=+37.182321897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert") pod "ingress-canary-4qp54" (UID: "0cfdbcdd-0be7-42e1-91eb-9892ee0ea925") : secret "canary-serving-cert" not found
May 20 13:50:55.648240 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:55.648202    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-th9gl" event={"ID":"c8b51a35-4910-4bcf-9198-c36444ad8dd5","Type":"ContainerStarted","Data":"f6b6431ebb1f49ca4d900302ea7e492a8d2594837a4efd203988d87b1cae5ff9"}
May 20 13:50:55.673963 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:55.673912    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-th9gl" podStartSLOduration=5.356203857 podStartE2EDuration="35.673896316s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:22.922963312 +0000 UTC m=+3.060517508" lastFinishedPulling="2026-05-20 13:50:53.240655771 +0000 UTC m=+33.378209967" observedRunningTime="2026-05-20 13:50:55.672041292 +0000 UTC m=+35.809595512" watchObservedRunningTime="2026-05-20 13:50:55.673896316 +0000 UTC m=+35.811450535"
May 20 13:50:57.059725 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:57.059686    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:50:57.060184 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:57.059796    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:50:57.060184 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:57.059803    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:57.060184 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:57.059896    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls podName:ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:01.059875685 +0000 UTC m=+41.197429885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls") pod "dns-default-8vcdb" (UID: "ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75") : secret "dns-default-metrics-tls" not found
May 20 13:50:57.060184 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:57.059911    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:57.060184 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:50:57.059951    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert podName:0cfdbcdd-0be7-42e1-91eb-9892ee0ea925 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:01.059940948 +0000 UTC m=+41.197495144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert") pod "ingress-canary-4qp54" (UID: "0cfdbcdd-0be7-42e1-91eb-9892ee0ea925") : secret "canary-serving-cert" not found
May 20 13:50:58.655156 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:58.655122    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wzjlt" event={"ID":"36b3619d-2709-46bd-af24-93312d906c20","Type":"ContainerStarted","Data":"f8efda990dd902281b9599133ff13ae55ebf327a68acb5be0d49c8ff90b22777"}
May 20 13:50:58.655620 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:58.655257    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:50:58.672129 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:50:58.672077    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wzjlt" podStartSLOduration=35.538296915 podStartE2EDuration="38.672062333s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:50:54.51518101 +0000 UTC m=+34.652735207" lastFinishedPulling="2026-05-20 13:50:57.648946426 +0000 UTC m=+37.786500625" observedRunningTime="2026-05-20 13:50:58.671162715 +0000 UTC m=+38.808716929" watchObservedRunningTime="2026-05-20 13:50:58.672062333 +0000 UTC m=+38.809616547"
May 20 13:51:01.084117 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:01.084088    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:51:01.084117 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:01.084122    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:51:01.084611 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:01.084212    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:01.084611 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:01.084221    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:01.084611 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:01.084262    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls podName:ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:09.084246617 +0000 UTC m=+49.221800813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls") pod "dns-default-8vcdb" (UID: "ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75") : secret "dns-default-metrics-tls" not found
May 20 13:51:01.084611 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:01.084275    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert podName:0cfdbcdd-0be7-42e1-91eb-9892ee0ea925 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:09.084268954 +0000 UTC m=+49.221823150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert") pod "ingress-canary-4qp54" (UID: "0cfdbcdd-0be7-42e1-91eb-9892ee0ea925") : secret "canary-serving-cert" not found
May 20 13:51:09.135751 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:09.135704    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:51:09.135751 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:09.135743    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:51:09.136383 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:09.135870    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:09.136383 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:09.135928    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:09.136383 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:09.135949    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert podName:0cfdbcdd-0be7-42e1-91eb-9892ee0ea925 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:25.135928409 +0000 UTC m=+65.273482610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert") pod "ingress-canary-4qp54" (UID: "0cfdbcdd-0be7-42e1-91eb-9892ee0ea925") : secret "canary-serving-cert" not found
May 20 13:51:09.136383 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:09.135988    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls podName:ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:25.135971639 +0000 UTC m=+65.273525864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls") pod "dns-default-8vcdb" (UID: "ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75") : secret "dns-default-metrics-tls" not found
May 20 13:51:18.636008 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:18.635977    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv7zm"
May 20 13:51:25.231421 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:25.231375    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:51:25.231421 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:25.231422    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:51:25.231911 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:25.231529    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:25.231911 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:25.231534    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:25.231911 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:25.231584    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls podName:ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:57.231568698 +0000 UTC m=+97.369122894 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls") pod "dns-default-8vcdb" (UID: "ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75") : secret "dns-default-metrics-tls" not found
May 20 13:51:25.231911 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:25.231598    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert podName:0cfdbcdd-0be7-42e1-91eb-9892ee0ea925 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:57.231592388 +0000 UTC m=+97.369146585 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert") pod "ingress-canary-4qp54" (UID: "0cfdbcdd-0be7-42e1-91eb-9892ee0ea925") : secret "canary-serving-cert" not found
May 20 13:51:26.237331 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:26.237291    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:51:26.240409 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:26.240385    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:51:26.247908 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:26.247886    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 13:51:26.247986 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:26.247943    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs podName:438841d1-e772-4acf-b0a6-58e5afe1937f nodeName:}" failed. No retries permitted until 2026-05-20 13:52:30.247925625 +0000 UTC m=+130.385479888 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs") pod "network-metrics-daemon-p9kp9" (UID: "438841d1-e772-4acf-b0a6-58e5afe1937f") : secret "metrics-daemon-secret" not found
May 20 13:51:29.659817 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:29.659788    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wzjlt"
May 20 13:51:57.330782 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:57.330752    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:51:57.331165 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:51:57.330792    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:51:57.331165 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:57.330907    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:57.331165 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:57.330910    2557 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:57.331165 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:57.330974    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls podName:ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:01.330958848 +0000 UTC m=+161.468513044 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls") pod "dns-default-8vcdb" (UID: "ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75") : secret "dns-default-metrics-tls" not found
May 20 13:51:57.331165 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:51:57.330989    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert podName:0cfdbcdd-0be7-42e1-91eb-9892ee0ea925 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:01.330982632 +0000 UTC m=+161.468536828 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert") pod "ingress-canary-4qp54" (UID: "0cfdbcdd-0be7-42e1-91eb-9892ee0ea925") : secret "canary-serving-cert" not found
May 20 13:52:03.616555 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.616521    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"]
May 20 13:52:03.618260 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.618241    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:03.624304 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.624266    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75d66f44d8-tqrhq"]
May 20 13:52:03.624979 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.624952    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 20 13:52:03.625434 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.625272    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 20 13:52:03.625434 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.625341    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-brzj6\""
May 20 13:52:03.625790 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.625763    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 20 13:52:03.626513 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.626340    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp"]
May 20 13:52:03.626746 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.626724    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.628159 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.628134    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-vgn62"]
May 20 13:52:03.628290 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.628269    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp"
May 20 13:52:03.629187 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.629169    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-42vs6\""
May 20 13:52:03.629905 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.629885    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.630009 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.629953    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 20 13:52:03.630248 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.630229    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 20 13:52:03.630985 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.630890    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 20 13:52:03.631448 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.631421    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 20 13:52:03.631653 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.631630    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 20 13:52:03.631818 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.631794    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 20 13:52:03.636025 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.636001    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 20 13:52:03.636135 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.636061    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9c45j\""
May 20 13:52:03.636196 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.636181    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"]
May 20 13:52:03.636685 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.636661    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 20 13:52:03.636784 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.636725    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 20 13:52:03.638000 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.637765    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 20 13:52:03.638417 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.638393    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ptrqp\""
May 20 13:52:03.639131 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.639111    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 20 13:52:03.639753 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.639733    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 20 13:52:03.648833 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.648807    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 20 13:52:03.648972 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.648870    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp"]
May 20 13:52:03.655117 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.655090    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-vgn62"]
May 20 13:52:03.659671 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.659651    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75d66f44d8-tqrhq"]
May 20 13:52:03.670699 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670680    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-config\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.670807 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670716    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:03.670807 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.670807 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670759    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-stats-auth\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.670807 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmdl\" (UniqueName: \"kubernetes.io/projected/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-kube-api-access-dpmdl\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.670964 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-trusted-ca\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.670964 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-serving-cert\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.670964 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670913    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9fh\" (UniqueName: \"kubernetes.io/projected/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-kube-api-access-sr9fh\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.670964 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670930    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-default-certificate\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.670964 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.671103 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.670988    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78pvq\" (UniqueName: \"kubernetes.io/projected/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-kube-api-access-78pvq\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:03.671103 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.671004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6kt7\" (UniqueName: \"kubernetes.io/projected/02e0e4f9-5851-4545-9e74-6782b356740c-kube-api-access-w6kt7\") pod \"volume-data-source-validator-5cf8c89d64-bkvlp\" (UID: \"02e0e4f9-5851-4545-9e74-6782b356740c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp"
May 20 13:52:03.722040 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.722020    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"]
May 20 13:52:03.723824 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.723810    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.727591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.727577    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 20 13:52:03.727711 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.727690    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 20 13:52:03.727768 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.727721    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 20 13:52:03.727810 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.727767    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 20 13:52:03.727881 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.727847    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9q9vp\""
May 20 13:52:03.740125 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.740103    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"]
May 20 13:52:03.771326 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771306    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:03.771405 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771333    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.771405 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42b21860-0b33-45ef-b866-d55adf52bbc8-config\") pod \"kube-storage-version-migrator-operator-565644f49-s25dc\" (UID: \"42b21860-0b33-45ef-b866-d55adf52bbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.771405 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771375    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-stats-auth\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.771405 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771390    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmdl\" (UniqueName: \"kubernetes.io/projected/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-kube-api-access-dpmdl\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.771405 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771406    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-trusted-ca\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.771605 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771453    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-serving-cert\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.771605 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771486    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgqt\" (UniqueName: \"kubernetes.io/projected/42b21860-0b33-45ef-b866-d55adf52bbc8-kube-api-access-dsgqt\") pod \"kube-storage-version-migrator-operator-565644f49-s25dc\" (UID: \"42b21860-0b33-45ef-b866-d55adf52bbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.771605 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:03.771459    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:03.771605 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:03.771521    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:04.271508138 +0000 UTC m=+104.409062383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : configmap references non-existent config key: service-ca.crt
May 20 13:52:03.771605 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771593    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9fh\" (UniqueName: \"kubernetes.io/projected/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-kube-api-access-sr9fh\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.771993 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:03.771689    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls podName:1f68f92c-cf6f-4b38-a13e-7c5a84214ee8 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:04.271672214 +0000 UTC m=+104.409226416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls") pod "cluster-samples-operator-697587599-l85dm" (UID: "1f68f92c-cf6f-4b38-a13e-7c5a84214ee8") : secret "samples-operator-tls" not found
May 20 13:52:03.771993 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771727    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-default-certificate\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.771993 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771752    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.771993 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771796    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78pvq\" (UniqueName: \"kubernetes.io/projected/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-kube-api-access-78pvq\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:03.771993 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771822    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6kt7\" (UniqueName: \"kubernetes.io/projected/02e0e4f9-5851-4545-9e74-6782b356740c-kube-api-access-w6kt7\") pod \"volume-data-source-validator-5cf8c89d64-bkvlp\" (UID: \"02e0e4f9-5851-4545-9e74-6782b356740c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp"
May 20 13:52:03.771993 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:03.771879    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:03.771993 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:03.771940    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:04.271922647 +0000 UTC m=+104.409476844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : secret "router-metrics-certs-default" not found
May 20 13:52:03.771993 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771882    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-config\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.772376 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.771999    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42b21860-0b33-45ef-b866-d55adf52bbc8-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-s25dc\" (UID: \"42b21860-0b33-45ef-b866-d55adf52bbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.772535 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.772511    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-trusted-ca\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.773311 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.773290    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-config\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.774941 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.774923    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-serving-cert\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.775021 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.775002    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-stats-auth\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.775156 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.775137    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-default-certificate\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.782111 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.782091    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78pvq\" (UniqueName: \"kubernetes.io/projected/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-kube-api-access-78pvq\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:03.782231 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.782215    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6kt7\" (UniqueName: \"kubernetes.io/projected/02e0e4f9-5851-4545-9e74-6782b356740c-kube-api-access-w6kt7\") pod \"volume-data-source-validator-5cf8c89d64-bkvlp\" (UID: \"02e0e4f9-5851-4545-9e74-6782b356740c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp"
May 20 13:52:03.782390 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.782376    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9fh\" (UniqueName: \"kubernetes.io/projected/9ed23ead-6ce7-4b98-a8f3-042a5d72979c-kube-api-access-sr9fh\") pod \"console-operator-76dff48d59-vgn62\" (UID: \"9ed23ead-6ce7-4b98-a8f3-042a5d72979c\") " pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:03.782756 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.782743    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmdl\" (UniqueName: \"kubernetes.io/projected/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-kube-api-access-dpmdl\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:03.872768 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.872721    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42b21860-0b33-45ef-b866-d55adf52bbc8-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-s25dc\" (UID: \"42b21860-0b33-45ef-b866-d55adf52bbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.872840 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.872774    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42b21860-0b33-45ef-b866-d55adf52bbc8-config\") pod \"kube-storage-version-migrator-operator-565644f49-s25dc\" (UID: \"42b21860-0b33-45ef-b866-d55adf52bbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.872840 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.872804    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgqt\" (UniqueName: \"kubernetes.io/projected/42b21860-0b33-45ef-b866-d55adf52bbc8-kube-api-access-dsgqt\") pod \"kube-storage-version-migrator-operator-565644f49-s25dc\" (UID: \"42b21860-0b33-45ef-b866-d55adf52bbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.873230 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.873212    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42b21860-0b33-45ef-b866-d55adf52bbc8-config\") pod \"kube-storage-version-migrator-operator-565644f49-s25dc\" (UID: \"42b21860-0b33-45ef-b866-d55adf52bbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.874546 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.874519    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42b21860-0b33-45ef-b866-d55adf52bbc8-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-s25dc\" (UID: \"42b21860-0b33-45ef-b866-d55adf52bbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.884763 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.884736    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgqt\" (UniqueName: \"kubernetes.io/projected/42b21860-0b33-45ef-b866-d55adf52bbc8-kube-api-access-dsgqt\") pod \"kube-storage-version-migrator-operator-565644f49-s25dc\" (UID: \"42b21860-0b33-45ef-b866-d55adf52bbc8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:03.949369 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.949348    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp"
May 20 13:52:03.953952 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:03.953932    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:04.032779 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.032389    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"
May 20 13:52:04.108590 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.108559    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-vgn62"]
May 20 13:52:04.111982 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:04.111952    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed23ead_6ce7_4b98_a8f3_042a5d72979c.slice/crio-5ecbf43c99b73963b30ebc53ee5130af39f5d64a828b570f9fb58ac19a393c4c WatchSource:0}: Error finding container 5ecbf43c99b73963b30ebc53ee5130af39f5d64a828b570f9fb58ac19a393c4c: Status 404 returned error can't find the container with id 5ecbf43c99b73963b30ebc53ee5130af39f5d64a828b570f9fb58ac19a393c4c
May 20 13:52:04.126390 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.126328    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp"]
May 20 13:52:04.128967 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:04.128937    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e0e4f9_5851_4545_9e74_6782b356740c.slice/crio-f3efada4318d3e60b894f4861df1f8166a422bcfd5142cd95c981a7c179531da WatchSource:0}: Error finding container f3efada4318d3e60b894f4861df1f8166a422bcfd5142cd95c981a7c179531da: Status 404 returned error can't find the container with id f3efada4318d3e60b894f4861df1f8166a422bcfd5142cd95c981a7c179531da
May 20 13:52:04.164997 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.164974    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc"]
May 20 13:52:04.167570 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:04.167545    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b21860_0b33_45ef_b866_d55adf52bbc8.slice/crio-bbb3b55ab9cbaf0125353959a89bfbbbf541fdcac4bd84c07fa8b7612221c727 WatchSource:0}: Error finding container bbb3b55ab9cbaf0125353959a89bfbbbf541fdcac4bd84c07fa8b7612221c727: Status 404 returned error can't find the container with id bbb3b55ab9cbaf0125353959a89bfbbbf541fdcac4bd84c07fa8b7612221c727
May 20 13:52:04.276065 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.276041    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:04.276162 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.276102    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:04.276162 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.276125    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:04.276272 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:04.276201    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:04.276272 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:04.276237    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:05.276223946 +0000 UTC m=+105.413778142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : configmap references non-existent config key: service-ca.crt
May 20 13:52:04.276272 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:04.276248    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:04.276272 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:04.276264    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:05.276245882 +0000 UTC m=+105.413800090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : secret "router-metrics-certs-default" not found
May 20 13:52:04.276418 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:04.276290    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls podName:1f68f92c-cf6f-4b38-a13e-7c5a84214ee8 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:05.276278272 +0000 UTC m=+105.413832469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls") pod "cluster-samples-operator-697587599-l85dm" (UID: "1f68f92c-cf6f-4b38-a13e-7c5a84214ee8") : secret "samples-operator-tls" not found
May 20 13:52:04.774525 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.774485    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" event={"ID":"9ed23ead-6ce7-4b98-a8f3-042a5d72979c","Type":"ContainerStarted","Data":"5ecbf43c99b73963b30ebc53ee5130af39f5d64a828b570f9fb58ac19a393c4c"}
May 20 13:52:04.775875 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.775814    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp" event={"ID":"02e0e4f9-5851-4545-9e74-6782b356740c","Type":"ContainerStarted","Data":"f3efada4318d3e60b894f4861df1f8166a422bcfd5142cd95c981a7c179531da"}
May 20 13:52:04.777873 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:04.777819    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc" event={"ID":"42b21860-0b33-45ef-b866-d55adf52bbc8","Type":"ContainerStarted","Data":"bbb3b55ab9cbaf0125353959a89bfbbbf541fdcac4bd84c07fa8b7612221c727"}
May 20 13:52:05.283474 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:05.283318    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:05.283474 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:05.283376    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:05.283698 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:05.283445    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:05.283698 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:05.283516    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:05.283698 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:05.283595    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls podName:1f68f92c-cf6f-4b38-a13e-7c5a84214ee8 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:07.283573469 +0000 UTC m=+107.421127678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls") pod "cluster-samples-operator-697587599-l85dm" (UID: "1f68f92c-cf6f-4b38-a13e-7c5a84214ee8") : secret "samples-operator-tls" not found
May 20 13:52:05.283698 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:05.283614    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:05.283698 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:05.283616    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:07.283606441 +0000 UTC m=+107.421160637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : configmap references non-existent config key: service-ca.crt
May 20 13:52:05.283698 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:05.283690    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:07.283671944 +0000 UTC m=+107.421226155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : secret "router-metrics-certs-default" not found
May 20 13:52:06.782809 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.782775    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp" event={"ID":"02e0e4f9-5851-4545-9e74-6782b356740c","Type":"ContainerStarted","Data":"0b825ad7e686185662dd362d8dff6124aba3801cc5c16f7fba3b88076ee48772"}
May 20 13:52:06.784115 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.784084    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc" event={"ID":"42b21860-0b33-45ef-b866-d55adf52bbc8","Type":"ContainerStarted","Data":"4ef3b1902b4b01cd48e052cd4172ed4095f597a397a17fde88c0b68f3153f6c8"}
May 20 13:52:06.785294 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.785277    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/0.log"
May 20 13:52:06.785389 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.785308    2557 generic.go:358] "Generic (PLEG): container finished" podID="9ed23ead-6ce7-4b98-a8f3-042a5d72979c" containerID="86caf011f3ad3f0abd8074afe7a7cf7903e31c3d8ed01d2323d88ce68fca8843" exitCode=255
May 20 13:52:06.785389 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.785337    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" event={"ID":"9ed23ead-6ce7-4b98-a8f3-042a5d72979c","Type":"ContainerDied","Data":"86caf011f3ad3f0abd8074afe7a7cf7903e31c3d8ed01d2323d88ce68fca8843"}
May 20 13:52:06.785565 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.785551    2557 scope.go:117] "RemoveContainer" containerID="86caf011f3ad3f0abd8074afe7a7cf7903e31c3d8ed01d2323d88ce68fca8843"
May 20 13:52:06.798849 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.798783    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-bkvlp" podStartSLOduration=1.861826524 podStartE2EDuration="3.798773649s" podCreationTimestamp="2026-05-20 13:52:03 +0000 UTC" firstStartedPulling="2026-05-20 13:52:04.130711579 +0000 UTC m=+104.268265775" lastFinishedPulling="2026-05-20 13:52:06.067658694 +0000 UTC m=+106.205212900" observedRunningTime="2026-05-20 13:52:06.798377977 +0000 UTC m=+106.935932197" watchObservedRunningTime="2026-05-20 13:52:06.798773649 +0000 UTC m=+106.936327866"
May 20 13:52:06.831381 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.831343    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc" podStartSLOduration=1.423396553 podStartE2EDuration="3.831332117s" podCreationTimestamp="2026-05-20 13:52:03 +0000 UTC" firstStartedPulling="2026-05-20 13:52:04.16909802 +0000 UTC m=+104.306652220" lastFinishedPulling="2026-05-20 13:52:06.577033588 +0000 UTC m=+106.714587784" observedRunningTime="2026-05-20 13:52:06.831183145 +0000 UTC m=+106.968737363" watchObservedRunningTime="2026-05-20 13:52:06.831332117 +0000 UTC m=+106.968886334"
May 20 13:52:06.993242 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.993207    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w"]
May 20 13:52:06.994980 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.994960    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w"
May 20 13:52:06.997634 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:06.997615    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qhsrc\""
May 20 13:52:07.003238 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.003220    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w"]
May 20 13:52:07.098349 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.098326    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcxc\" (UniqueName: \"kubernetes.io/projected/af961ba2-f410-45db-84ef-e8d1c983ed54-kube-api-access-9fcxc\") pod \"network-check-source-7f44b6dd87-tz48w\" (UID: \"af961ba2-f410-45db-84ef-e8d1c983ed54\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w"
May 20 13:52:07.199361 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.199342    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcxc\" (UniqueName: \"kubernetes.io/projected/af961ba2-f410-45db-84ef-e8d1c983ed54-kube-api-access-9fcxc\") pod \"network-check-source-7f44b6dd87-tz48w\" (UID: \"af961ba2-f410-45db-84ef-e8d1c983ed54\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w"
May 20 13:52:07.208934 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.208913    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcxc\" (UniqueName: \"kubernetes.io/projected/af961ba2-f410-45db-84ef-e8d1c983ed54-kube-api-access-9fcxc\") pod \"network-check-source-7f44b6dd87-tz48w\" (UID: \"af961ba2-f410-45db-84ef-e8d1c983ed54\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w"
May 20 13:52:07.300447 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.300420    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:07.300543 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.300452    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:07.300543 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.300486    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:07.300649 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:07.300555    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:07.300649 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:07.300560    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:07.300649 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:07.300598    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:11.300579516 +0000 UTC m=+111.438133724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : configmap references non-existent config key: service-ca.crt
May 20 13:52:07.300649 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:07.300624    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls podName:1f68f92c-cf6f-4b38-a13e-7c5a84214ee8 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:11.300613988 +0000 UTC m=+111.438168201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls") pod "cluster-samples-operator-697587599-l85dm" (UID: "1f68f92c-cf6f-4b38-a13e-7c5a84214ee8") : secret "samples-operator-tls" not found
May 20 13:52:07.300649 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:07.300639    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:11.30063055 +0000 UTC m=+111.438184753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : secret "router-metrics-certs-default" not found
May 20 13:52:07.302735 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.302719    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w"
May 20 13:52:07.424096 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.424067    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w"]
May 20 13:52:07.427490 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:07.427455    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf961ba2_f410_45db_84ef_e8d1c983ed54.slice/crio-30c9684565fc9de1e85859fa8f075a4e47ca1dfe41645afd34bd351db53a88ef WatchSource:0}: Error finding container 30c9684565fc9de1e85859fa8f075a4e47ca1dfe41645afd34bd351db53a88ef: Status 404 returned error can't find the container with id 30c9684565fc9de1e85859fa8f075a4e47ca1dfe41645afd34bd351db53a88ef
May 20 13:52:07.628225 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.628146    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2"]
May 20 13:52:07.630459 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.630438    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2"
May 20 13:52:07.633562 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.633539    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 20 13:52:07.633667 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.633576    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lkdh6\""
May 20 13:52:07.633924 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.633906    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 20 13:52:07.651040 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.651009    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2"]
May 20 13:52:07.703813 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.703794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4gz\" (UniqueName: \"kubernetes.io/projected/1b095c39-238f-4149-92de-db3dc849e556-kube-api-access-sf4gz\") pod \"migrator-b985f5d89-s4nl2\" (UID: \"1b095c39-238f-4149-92de-db3dc849e556\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2"
May 20 13:52:07.788989 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.788959    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w" event={"ID":"af961ba2-f410-45db-84ef-e8d1c983ed54","Type":"ContainerStarted","Data":"214d6969496670870f81893d71e4ff78e72bb827430c17e6717c9aefb75c9f49"}
May 20 13:52:07.789298 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.788993    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w" event={"ID":"af961ba2-f410-45db-84ef-e8d1c983ed54","Type":"ContainerStarted","Data":"30c9684565fc9de1e85859fa8f075a4e47ca1dfe41645afd34bd351db53a88ef"}
May 20 13:52:07.790309 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.790293    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/1.log"
May 20 13:52:07.790639 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.790626    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/0.log"
May 20 13:52:07.790717 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.790654    2557 generic.go:358] "Generic (PLEG): container finished" podID="9ed23ead-6ce7-4b98-a8f3-042a5d72979c" containerID="02f51542baa94ee807d32ab2eb9432803b659143c01150047badfdb42bdc0d63" exitCode=255
May 20 13:52:07.790773 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.790741    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" event={"ID":"9ed23ead-6ce7-4b98-a8f3-042a5d72979c","Type":"ContainerDied","Data":"02f51542baa94ee807d32ab2eb9432803b659143c01150047badfdb42bdc0d63"}
May 20 13:52:07.790813 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.790773    2557 scope.go:117] "RemoveContainer" containerID="86caf011f3ad3f0abd8074afe7a7cf7903e31c3d8ed01d2323d88ce68fca8843"
May 20 13:52:07.791071 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.791051    2557 scope.go:117] "RemoveContainer" containerID="02f51542baa94ee807d32ab2eb9432803b659143c01150047badfdb42bdc0d63"
May 20 13:52:07.791252 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:07.791231    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-vgn62_openshift-console-operator(9ed23ead-6ce7-4b98-a8f3-042a5d72979c)\"" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" podUID="9ed23ead-6ce7-4b98-a8f3-042a5d72979c"
May 20 13:52:07.804174 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.804137    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4gz\" (UniqueName: \"kubernetes.io/projected/1b095c39-238f-4149-92de-db3dc849e556-kube-api-access-sf4gz\") pod \"migrator-b985f5d89-s4nl2\" (UID: \"1b095c39-238f-4149-92de-db3dc849e556\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2"
May 20 13:52:07.805270 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.805237    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tz48w" podStartSLOduration=1.805227759 podStartE2EDuration="1.805227759s" podCreationTimestamp="2026-05-20 13:52:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:52:07.80436954 +0000 UTC m=+107.941923756" watchObservedRunningTime="2026-05-20 13:52:07.805227759 +0000 UTC m=+107.942781977"
May 20 13:52:07.818116 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.818094    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4gz\" (UniqueName: \"kubernetes.io/projected/1b095c39-238f-4149-92de-db3dc849e556-kube-api-access-sf4gz\") pod \"migrator-b985f5d89-s4nl2\" (UID: \"1b095c39-238f-4149-92de-db3dc849e556\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2"
May 20 13:52:07.939335 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:07.939289    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2"
May 20 13:52:08.056136 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:08.056109    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2"]
May 20 13:52:08.059084 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:08.059056    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b095c39_238f_4149_92de_db3dc849e556.slice/crio-8052af25eef207c02de85f7110b8b0851aa0209673ef900f114c73476397a119 WatchSource:0}: Error finding container 8052af25eef207c02de85f7110b8b0851aa0209673ef900f114c73476397a119: Status 404 returned error can't find the container with id 8052af25eef207c02de85f7110b8b0851aa0209673ef900f114c73476397a119
May 20 13:52:08.797709 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:08.797678    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/1.log"
May 20 13:52:08.798219 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:08.798041    2557 scope.go:117] "RemoveContainer" containerID="02f51542baa94ee807d32ab2eb9432803b659143c01150047badfdb42bdc0d63"
May 20 13:52:08.798273 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:08.798239    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-vgn62_openshift-console-operator(9ed23ead-6ce7-4b98-a8f3-042a5d72979c)\"" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" podUID="9ed23ead-6ce7-4b98-a8f3-042a5d72979c"
May 20 13:52:08.798907 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:08.798879    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2" event={"ID":"1b095c39-238f-4149-92de-db3dc849e556","Type":"ContainerStarted","Data":"8052af25eef207c02de85f7110b8b0851aa0209673ef900f114c73476397a119"}
May 20 13:52:09.386760 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:09.386738    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-58wjx_81d61543-18a2-43a7-b322-54bdaf92f4a9/dns-node-resolver/0.log"
May 20 13:52:09.808234 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:09.808196    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2" event={"ID":"1b095c39-238f-4149-92de-db3dc849e556","Type":"ContainerStarted","Data":"26134f26ea4b0467b81154c33102ba02719e249ab45122bf4dc5c9680fd7a90d"}
May 20 13:52:09.808234 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:09.808236    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2" event={"ID":"1b095c39-238f-4149-92de-db3dc849e556","Type":"ContainerStarted","Data":"41955df38bb523f18f954a0ea3d11bb50860586b4e1ee9a52bd7a0f5df859fe2"}
May 20 13:52:09.829145 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:09.829101    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-s4nl2" podStartSLOduration=1.7194008950000002 podStartE2EDuration="2.829086236s" podCreationTimestamp="2026-05-20 13:52:07 +0000 UTC" firstStartedPulling="2026-05-20 13:52:08.060989681 +0000 UTC m=+108.198543878" lastFinishedPulling="2026-05-20 13:52:09.170675018 +0000 UTC m=+109.308229219" observedRunningTime="2026-05-20 13:52:09.82780306 +0000 UTC m=+109.965357278" watchObservedRunningTime="2026-05-20 13:52:09.829086236 +0000 UTC m=+109.966640453"
May 20 13:52:10.775800 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:10.775774    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ww2l9_622157d3-3855-4995-8334-d8407eaffade/node-ca/0.log"
May 20 13:52:11.330752 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:11.330702    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:11.331149 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:11.330812    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:11.331149 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:11.330840    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:11.331149 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:11.330870    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:11.331149 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:11.330916    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:19.330900975 +0000 UTC m=+119.468455172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : secret "router-metrics-certs-default" not found
May 20 13:52:11.331149 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:11.330955    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:11.331149 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:11.330994    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:19.330976856 +0000 UTC m=+119.468531086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : configmap references non-existent config key: service-ca.crt
May 20 13:52:11.331149 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:11.331024    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls podName:1f68f92c-cf6f-4b38-a13e-7c5a84214ee8 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:19.331011665 +0000 UTC m=+119.468565920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls") pod "cluster-samples-operator-697587599-l85dm" (UID: "1f68f92c-cf6f-4b38-a13e-7c5a84214ee8") : secret "samples-operator-tls" not found
May 20 13:52:13.954776 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:13.954732    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:13.954776 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:13.954785    2557 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:13.955219 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:13.955159    2557 scope.go:117] "RemoveContainer" containerID="02f51542baa94ee807d32ab2eb9432803b659143c01150047badfdb42bdc0d63"
May 20 13:52:13.955358 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:13.955337    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-vgn62_openshift-console-operator(9ed23ead-6ce7-4b98-a8f3-042a5d72979c)\"" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" podUID="9ed23ead-6ce7-4b98-a8f3-042a5d72979c"
May 20 13:52:19.393282 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:19.393232    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:19.393677 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:19.393339    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:19.393677 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:19.393377    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:19.393677 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:19.393545    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle podName:72fcf5ad-c6b9-47b9-a98e-25767a7ead5a nodeName:}" failed. No retries permitted until 2026-05-20 13:52:35.393522448 +0000 UTC m=+135.531076673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle") pod "router-default-75d66f44d8-tqrhq" (UID: "72fcf5ad-c6b9-47b9-a98e-25767a7ead5a") : configmap references non-existent config key: service-ca.crt
May 20 13:52:19.395894 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:19.395848    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-metrics-certs\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:19.395973 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:19.395915    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1f68f92c-cf6f-4b38-a13e-7c5a84214ee8-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l85dm\" (UID: \"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:19.529342 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:19.529315    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"
May 20 13:52:19.651214 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:19.651155    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm"]
May 20 13:52:19.833609 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:19.833576    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm" event={"ID":"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8","Type":"ContainerStarted","Data":"345947eff3e675c515fd1f9a2c421952b95f591dd86df8d0b8d427630c5a21c9"}
May 20 13:52:21.842419 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:21.842374    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm" event={"ID":"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8","Type":"ContainerStarted","Data":"ec0a2905f74e3cd66fa61bd6762fde629a8f0639245b8a3a03f53e2a0e81567d"}
May 20 13:52:21.842419 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:21.842419    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm" event={"ID":"1f68f92c-cf6f-4b38-a13e-7c5a84214ee8","Type":"ContainerStarted","Data":"ed8827af26c174cb9e8e7507308345e1eafec78f0f79c4a633e493120a3c240e"}
May 20 13:52:25.401278 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:25.401246    2557 scope.go:117] "RemoveContainer" containerID="02f51542baa94ee807d32ab2eb9432803b659143c01150047badfdb42bdc0d63"
May 20 13:52:25.858723 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:25.858694    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 13:52:25.859139 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:25.859119    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/1.log"
May 20 13:52:25.859201 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:25.859159    2557 generic.go:358] "Generic (PLEG): container finished" podID="9ed23ead-6ce7-4b98-a8f3-042a5d72979c" containerID="ce6194e3f1dcc178003421d3860724976c53bbe35907c6a3c631d621d9535930" exitCode=255
May 20 13:52:25.859238 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:25.859203    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" event={"ID":"9ed23ead-6ce7-4b98-a8f3-042a5d72979c","Type":"ContainerDied","Data":"ce6194e3f1dcc178003421d3860724976c53bbe35907c6a3c631d621d9535930"}
May 20 13:52:25.859276 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:25.859240    2557 scope.go:117] "RemoveContainer" containerID="02f51542baa94ee807d32ab2eb9432803b659143c01150047badfdb42bdc0d63"
May 20 13:52:25.859602 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:25.859571    2557 scope.go:117] "RemoveContainer" containerID="ce6194e3f1dcc178003421d3860724976c53bbe35907c6a3c631d621d9535930"
May 20 13:52:25.859827 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:25.859785    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-vgn62_openshift-console-operator(9ed23ead-6ce7-4b98-a8f3-042a5d72979c)\"" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" podUID="9ed23ead-6ce7-4b98-a8f3-042a5d72979c"
May 20 13:52:25.877524 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:25.877481    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l85dm" podStartSLOduration=21.251595035 podStartE2EDuration="22.877468777s" podCreationTimestamp="2026-05-20 13:52:03 +0000 UTC" firstStartedPulling="2026-05-20 13:52:19.685771585 +0000 UTC m=+119.823325781" lastFinishedPulling="2026-05-20 13:52:21.311645318 +0000 UTC m=+121.449199523" observedRunningTime="2026-05-20 13:52:21.868051848 +0000 UTC m=+122.005606067" watchObservedRunningTime="2026-05-20 13:52:25.877468777 +0000 UTC m=+126.015022994"
May 20 13:52:26.862665 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:26.862645    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 13:52:30.266307 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:30.266268    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:52:30.268594 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:30.268571    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/438841d1-e772-4acf-b0a6-58e5afe1937f-metrics-certs\") pod \"network-metrics-daemon-p9kp9\" (UID: \"438841d1-e772-4acf-b0a6-58e5afe1937f\") " pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:52:30.415219 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:30.415191    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d6bxt\""
May 20 13:52:30.422104 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:30.422077    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9kp9"
May 20 13:52:30.547352 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:30.547295    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9kp9"]
May 20 13:52:30.550087 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:30.550060    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438841d1_e772_4acf_b0a6_58e5afe1937f.slice/crio-ca15feac334692aa353db1694c2f87c107b08bfe92e4f4498b0da1bbbbdceeec WatchSource:0}: Error finding container ca15feac334692aa353db1694c2f87c107b08bfe92e4f4498b0da1bbbbdceeec: Status 404 returned error can't find the container with id ca15feac334692aa353db1694c2f87c107b08bfe92e4f4498b0da1bbbbdceeec
May 20 13:52:30.872760 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:30.872727    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9kp9" event={"ID":"438841d1-e772-4acf-b0a6-58e5afe1937f","Type":"ContainerStarted","Data":"ca15feac334692aa353db1694c2f87c107b08bfe92e4f4498b0da1bbbbdceeec"}
May 20 13:52:31.233338 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.233263    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l65x8"]
May 20 13:52:31.235451 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.235433    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.238833 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.238735    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 13:52:31.240094 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.240072    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 13:52:31.240094 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.240087    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 13:52:31.240253 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.240228    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 13:52:31.240667 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.240647    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gcddw\""
May 20 13:52:31.254243 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.254221    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l65x8"]
May 20 13:52:31.375150 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.375116    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4da3b58f-efe6-42c5-baba-7d144e34ebef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.375529 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.375170    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4da3b58f-efe6-42c5-baba-7d144e34ebef-crio-socket\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.375529 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.375231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9ls\" (UniqueName: \"kubernetes.io/projected/4da3b58f-efe6-42c5-baba-7d144e34ebef-kube-api-access-jk9ls\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.375529 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.375275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4da3b58f-efe6-42c5-baba-7d144e34ebef-data-volume\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.375529 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.375300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4da3b58f-efe6-42c5-baba-7d144e34ebef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.475701 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.475671    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9ls\" (UniqueName: \"kubernetes.io/projected/4da3b58f-efe6-42c5-baba-7d144e34ebef-kube-api-access-jk9ls\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.475784 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.475756    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4da3b58f-efe6-42c5-baba-7d144e34ebef-data-volume\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.475838 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.475786    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4da3b58f-efe6-42c5-baba-7d144e34ebef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.475908 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.475843    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4da3b58f-efe6-42c5-baba-7d144e34ebef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.475908 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.475891    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4da3b58f-efe6-42c5-baba-7d144e34ebef-crio-socket\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.476057 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.475993    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4da3b58f-efe6-42c5-baba-7d144e34ebef-crio-socket\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.476134 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.476116    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4da3b58f-efe6-42c5-baba-7d144e34ebef-data-volume\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.476377 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.476346    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4da3b58f-efe6-42c5-baba-7d144e34ebef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.478070 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.478051    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4da3b58f-efe6-42c5-baba-7d144e34ebef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.486744 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.486695    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9ls\" (UniqueName: \"kubernetes.io/projected/4da3b58f-efe6-42c5-baba-7d144e34ebef-kube-api-access-jk9ls\") pod \"insights-runtime-extractor-l65x8\" (UID: \"4da3b58f-efe6-42c5-baba-7d144e34ebef\") " pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.550371 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.550348    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l65x8"
May 20 13:52:31.711350 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.711329    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l65x8"]
May 20 13:52:31.712866 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:31.712831    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da3b58f_efe6_42c5_baba_7d144e34ebef.slice/crio-5e19a7bbea7e16966d6763ba998745a9ab86dddd622155d67a1c9dbfeaa6b477 WatchSource:0}: Error finding container 5e19a7bbea7e16966d6763ba998745a9ab86dddd622155d67a1c9dbfeaa6b477: Status 404 returned error can't find the container with id 5e19a7bbea7e16966d6763ba998745a9ab86dddd622155d67a1c9dbfeaa6b477
May 20 13:52:31.877249 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.877215    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9kp9" event={"ID":"438841d1-e772-4acf-b0a6-58e5afe1937f","Type":"ContainerStarted","Data":"e623b76d436956962946a91859bd5c436ab96a32f21368c70ea5fb9b58e596cd"}
May 20 13:52:31.877249 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.877251    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9kp9" event={"ID":"438841d1-e772-4acf-b0a6-58e5afe1937f","Type":"ContainerStarted","Data":"120b9a457bfd125207043596270b8e24a4be475996db69011e8309beaa033619"}
May 20 13:52:31.878510 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.878486    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l65x8" event={"ID":"4da3b58f-efe6-42c5-baba-7d144e34ebef","Type":"ContainerStarted","Data":"7b47416db83070302e1cd33f8118ba3eca76c94a312f09b1b3b9bedfd3fa3a3b"}
May 20 13:52:31.878613 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.878514    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l65x8" event={"ID":"4da3b58f-efe6-42c5-baba-7d144e34ebef","Type":"ContainerStarted","Data":"5e19a7bbea7e16966d6763ba998745a9ab86dddd622155d67a1c9dbfeaa6b477"}
May 20 13:52:31.929393 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:31.929353    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p9kp9" podStartSLOduration=131.06563431 podStartE2EDuration="2m11.929339157s" podCreationTimestamp="2026-05-20 13:50:20 +0000 UTC" firstStartedPulling="2026-05-20 13:52:30.551919314 +0000 UTC m=+130.689473514" lastFinishedPulling="2026-05-20 13:52:31.415624161 +0000 UTC m=+131.553178361" observedRunningTime="2026-05-20 13:52:31.925192983 +0000 UTC m=+132.062747200" watchObservedRunningTime="2026-05-20 13:52:31.929339157 +0000 UTC m=+132.066893375"
May 20 13:52:32.887350 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:32.887319    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l65x8" event={"ID":"4da3b58f-efe6-42c5-baba-7d144e34ebef","Type":"ContainerStarted","Data":"881c1b8e9883f882653f80f554ac0dfddfa959e074a0ee8794d571a50b85a97f"}
May 20 13:52:33.893079 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:33.893054    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l65x8" event={"ID":"4da3b58f-efe6-42c5-baba-7d144e34ebef","Type":"ContainerStarted","Data":"37d168d904adf8b73e391e46a79aa92d182cdf77883825072d3cc2bdb2f5d37e"}
May 20 13:52:33.954147 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:33.954124    2557 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:33.954237 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:33.954170    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:33.954433 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:33.954419    2557 scope.go:117] "RemoveContainer" containerID="ce6194e3f1dcc178003421d3860724976c53bbe35907c6a3c631d621d9535930"
May 20 13:52:33.954572 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:33.954557    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-vgn62_openshift-console-operator(9ed23ead-6ce7-4b98-a8f3-042a5d72979c)\"" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" podUID="9ed23ead-6ce7-4b98-a8f3-042a5d72979c"
May 20 13:52:33.981842 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:33.981801    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l65x8" podStartSLOduration=0.931666788 podStartE2EDuration="2.981788121s" podCreationTimestamp="2026-05-20 13:52:31 +0000 UTC" firstStartedPulling="2026-05-20 13:52:31.763389439 +0000 UTC m=+131.900943635" lastFinishedPulling="2026-05-20 13:52:33.813510766 +0000 UTC m=+133.951064968" observedRunningTime="2026-05-20 13:52:33.922774967 +0000 UTC m=+134.060329183" watchObservedRunningTime="2026-05-20 13:52:33.981788121 +0000 UTC m=+134.119342342"
May 20 13:52:35.403246 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:35.403212    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:35.407562 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:35.404514    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72fcf5ad-c6b9-47b9-a98e-25767a7ead5a-service-ca-bundle\") pod \"router-default-75d66f44d8-tqrhq\" (UID: \"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a\") " pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:35.444223 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:35.444201    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-42vs6\""
May 20 13:52:35.452320 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:35.452301    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:35.586962 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:35.586940    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75d66f44d8-tqrhq"]
May 20 13:52:35.589166 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:35.589136    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fcf5ad_c6b9_47b9_a98e_25767a7ead5a.slice/crio-73000317987b06cf0ed2cba3cd97faa2808178a04b0c7f7deeee44dde2dbab20 WatchSource:0}: Error finding container 73000317987b06cf0ed2cba3cd97faa2808178a04b0c7f7deeee44dde2dbab20: Status 404 returned error can't find the container with id 73000317987b06cf0ed2cba3cd97faa2808178a04b0c7f7deeee44dde2dbab20
May 20 13:52:35.899522 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:35.899483    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75d66f44d8-tqrhq" event={"ID":"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a","Type":"ContainerStarted","Data":"eaa9f00b9ffee58b96b6568836069f2cfa343ae66659d6d2d4fa3c228339e9bf"}
May 20 13:52:35.899522 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:35.899522    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75d66f44d8-tqrhq" event={"ID":"72fcf5ad-c6b9-47b9-a98e-25767a7ead5a","Type":"ContainerStarted","Data":"73000317987b06cf0ed2cba3cd97faa2808178a04b0c7f7deeee44dde2dbab20"}
May 20 13:52:35.925473 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:35.925422    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-75d66f44d8-tqrhq" podStartSLOduration=32.925406517 podStartE2EDuration="32.925406517s" podCreationTimestamp="2026-05-20 13:52:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:52:35.924356402 +0000 UTC m=+136.061910620" watchObservedRunningTime="2026-05-20 13:52:35.925406517 +0000 UTC m=+136.062960737"
May 20 13:52:36.453213 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:36.453183    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:36.455812 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:36.455783    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:36.902790 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:36.902765    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:36.904184 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:36.904159    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75d66f44d8-tqrhq"
May 20 13:52:44.401079 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:44.401047    2557 scope.go:117] "RemoveContainer" containerID="ce6194e3f1dcc178003421d3860724976c53bbe35907c6a3c631d621d9535930"
May 20 13:52:44.401488 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:44.401238    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-vgn62_openshift-console-operator(9ed23ead-6ce7-4b98-a8f3-042a5d72979c)\"" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" podUID="9ed23ead-6ce7-4b98-a8f3-042a5d72979c"
May 20 13:52:47.461523 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.461487    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"]
May 20 13:52:47.465026 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.465008    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.469891 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.469867    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 13:52:47.470032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.469910    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 13:52:47.470032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.469926    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 20 13:52:47.470032 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.470004    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 13:52:47.470762 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.470744    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 20 13:52:47.471024 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.471004    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cmkdv\""
May 20 13:52:47.479435 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.479412    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"]
May 20 13:52:47.481376 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.481359    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.484918 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.484900    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z58fc\""
May 20 13:52:47.485235 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.485221    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 20 13:52:47.488282 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.488264    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"]
May 20 13:52:47.491476 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.491460    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 20 13:52:47.491563 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.491528    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 20 13:52:47.515877 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.514766    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"]
May 20 13:52:47.554070 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.554049    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sx46z"]
May 20 13:52:47.556056 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.556042    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.561182 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.561112    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 13:52:47.561787 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.561769    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 13:52:47.561931 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.561915    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-78qbf\""
May 20 13:52:47.561985 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.561916    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 13:52:47.585155 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585132    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.585231 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585160    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.585231 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585178    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.585231 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585201    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.585231 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj97p\" (UniqueName: \"kubernetes.io/projected/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-kube-api-access-gj97p\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.585372 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.585372 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppxh\" (UniqueName: \"kubernetes.io/projected/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-api-access-zppxh\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.585372 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585330    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.585372 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.585499 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.585378    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.686232 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686207    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.686303 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686241    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.686303 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686264    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.686303 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.686409 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686322    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.686409 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686370    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.686487 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bac2ac0-f861-498f-92f6-049402b8f6d6-metrics-client-ca\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.686487 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686444    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.686487 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686469    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gj97p\" (UniqueName: \"kubernetes.io/projected/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-kube-api-access-gj97p\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.686628 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686514    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.686781 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686745    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zppxh\" (UniqueName: \"kubernetes.io/projected/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-api-access-zppxh\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.686887 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686788    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bac2ac0-f861-498f-92f6-049402b8f6d6-sys\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.686887 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686818    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-textfile\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.686887 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-tls\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.687059 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686928    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.687059 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686950    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.687059 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686967    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.687059 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.686956    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bac2ac0-f861-498f-92f6-049402b8f6d6-root\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.687059 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.687034    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.687273 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.687059    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrpn\" (UniqueName: \"kubernetes.io/projected/9bac2ac0-f861-498f-92f6-049402b8f6d6-kube-api-access-hcrpn\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.687273 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.687096    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.687273 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.687128    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-wtmp\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.687273 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:47.687197    2557 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 20 13:52:47.687273 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:47.687270    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-tls podName:011b9ea8-4788-46a9-9cb6-d5c6f20b843d nodeName:}" failed. No retries permitted until 2026-05-20 13:52:48.18725177 +0000 UTC m=+148.324805978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-rvzhf" (UID: "011b9ea8-4788-46a9-9cb6-d5c6f20b843d") : secret "kube-state-metrics-tls" not found
May 20 13:52:47.687569 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.687551    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.688776 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.688748    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.689083 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.689065    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.689232 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.689212    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.697821 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.697787    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppxh\" (UniqueName: \"kubernetes.io/projected/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-api-access-zppxh\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:47.701217 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.701198    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj97p\" (UniqueName: \"kubernetes.io/projected/bb528ea5-f9cc-4ff8-91d2-49c5452bf8be-kube-api-access-gj97p\") pod \"openshift-state-metrics-9c959559f-6sb49\" (UID: \"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.774271 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.774210    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"
May 20 13:52:47.787973 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.787949    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-wtmp\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788059 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788004    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788059 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788025    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788161 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788138    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bac2ac0-f861-498f-92f6-049402b8f6d6-metrics-client-ca\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788216 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788195    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bac2ac0-f861-498f-92f6-049402b8f6d6-sys\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788268 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788222    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-textfile\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788268 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788249    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-tls\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788369 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788260    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bac2ac0-f861-498f-92f6-049402b8f6d6-sys\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788369 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788147    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-wtmp\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788369 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788293    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bac2ac0-f861-498f-92f6-049402b8f6d6-root\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788369 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788323    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrpn\" (UniqueName: \"kubernetes.io/projected/9bac2ac0-f861-498f-92f6-049402b8f6d6-kube-api-access-hcrpn\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788540 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788412    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9bac2ac0-f861-498f-92f6-049402b8f6d6-root\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788540 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788512    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-textfile\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788630 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788585    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-accelerators-collector-config\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.788756 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.788740    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bac2ac0-f861-498f-92f6-049402b8f6d6-metrics-client-ca\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.790493 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.790471    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.790634 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.790613    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9bac2ac0-f861-498f-92f6-049402b8f6d6-node-exporter-tls\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.798174 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.798151    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrpn\" (UniqueName: \"kubernetes.io/projected/9bac2ac0-f861-498f-92f6-049402b8f6d6-kube-api-access-hcrpn\") pod \"node-exporter-sx46z\" (UID: \"9bac2ac0-f861-498f-92f6-049402b8f6d6\") " pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.864309 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.863997    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sx46z"
May 20 13:52:47.872820 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:47.872783    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bac2ac0_f861_498f_92f6_049402b8f6d6.slice/crio-88d4401de97fbb59943361f3a00ec0765b027ac9c608e9c56ed30a598bf99484 WatchSource:0}: Error finding container 88d4401de97fbb59943361f3a00ec0765b027ac9c608e9c56ed30a598bf99484: Status 404 returned error can't find the container with id 88d4401de97fbb59943361f3a00ec0765b027ac9c608e9c56ed30a598bf99484
May 20 13:52:47.911776 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.911749    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-6sb49"]
May 20 13:52:47.914080 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:47.914058    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb528ea5_f9cc_4ff8_91d2_49c5452bf8be.slice/crio-e262783a67ab46363d5176c9fe774f6cb52b082dcd8c2f68b641a851c137898d WatchSource:0}: Error finding container e262783a67ab46363d5176c9fe774f6cb52b082dcd8c2f68b641a851c137898d: Status 404 returned error can't find the container with id e262783a67ab46363d5176c9fe774f6cb52b082dcd8c2f68b641a851c137898d
May 20 13:52:47.933649 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.933610    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49" event={"ID":"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be","Type":"ContainerStarted","Data":"e262783a67ab46363d5176c9fe774f6cb52b082dcd8c2f68b641a851c137898d"}
May 20 13:52:47.934662 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:47.934634    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sx46z" event={"ID":"9bac2ac0-f861-498f-92f6-049402b8f6d6","Type":"ContainerStarted","Data":"88d4401de97fbb59943361f3a00ec0765b027ac9c608e9c56ed30a598bf99484"}
May 20 13:52:48.190957 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:48.190929    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:48.193007 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:48.192990    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/011b9ea8-4788-46a9-9cb6-d5c6f20b843d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-rvzhf\" (UID: \"011b9ea8-4788-46a9-9cb6-d5c6f20b843d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:48.389482 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:48.389455    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"
May 20 13:52:48.578712 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:48.578685    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf"]
May 20 13:52:48.707422 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:48.707357    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011b9ea8_4788_46a9_9cb6_d5c6f20b843d.slice/crio-acf16a05b11e89f346b51177b0fdf8a18fe260a0a3945609d6a93c201b2d8006 WatchSource:0}: Error finding container acf16a05b11e89f346b51177b0fdf8a18fe260a0a3945609d6a93c201b2d8006: Status 404 returned error can't find the container with id acf16a05b11e89f346b51177b0fdf8a18fe260a0a3945609d6a93c201b2d8006
May 20 13:52:48.941610 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:48.941573    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf" event={"ID":"011b9ea8-4788-46a9-9cb6-d5c6f20b843d","Type":"ContainerStarted","Data":"acf16a05b11e89f346b51177b0fdf8a18fe260a0a3945609d6a93c201b2d8006"}
May 20 13:52:48.943411 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:48.943382    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49" event={"ID":"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be","Type":"ContainerStarted","Data":"e0f01279c7eeb4c9ec7ecf6ce0de94afafb603afa75057530aab54b1f9fcb720"}
May 20 13:52:48.943539 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:48.943420    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49" event={"ID":"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be","Type":"ContainerStarted","Data":"d8e15e337dd6a2734a0d1216bc05f99da88647a1761488324552f678208341b0"}
May 20 13:52:48.944786 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:48.944760    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sx46z" event={"ID":"9bac2ac0-f861-498f-92f6-049402b8f6d6","Type":"ContainerStarted","Data":"fa0e35dcf39096dc1bca78d1fd49d1d8c1a70239c8115966014ab4cb197da1ae"}
May 20 13:52:49.080389 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.080368    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:52:49.082901 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.082846    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.086179 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.086151    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 20 13:52:49.086259 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.086229    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 20 13:52:49.086509 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.086487    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 20 13:52:49.089185 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.089093    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 20 13:52:49.089185 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.089116    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 20 13:52:49.089185 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.089166    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 20 13:52:49.090218 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.090202    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7q8hw\""
May 20 13:52:49.090312 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.090237    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 20 13:52:49.093587 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.093565    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 20 13:52:49.094477 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.094463    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 20 13:52:49.116415 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.116391    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:52:49.200432 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200411    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200521 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200450    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-web-config\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200521 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200473    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200521 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200490    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200619 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200537    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200619 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200554    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-config-out\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200619 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200580    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200619 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jtt\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-kube-api-access-k2jtt\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200733 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200645    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200733 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200704    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200733 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200725    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-config-volume\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200818 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.200818 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.200770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.301955 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.301901    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.301955 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.301939    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jtt\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-kube-api-access-k2jtt\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302068 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.301970    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302144 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302123    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302182 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302167    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-config-volume\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302317 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302289    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302428 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302329    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302428 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302396    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302428 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302423    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-web-config\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302602 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302458    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302602 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302483    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302602 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302511    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.302602 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.302537    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-config-out\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.303097 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.303074    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.303850 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.303821    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.304146 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.304127    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.305064 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.305041    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.305316 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.305278    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-config-out\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.305403 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.305326    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-config-volume\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.305556 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.305538    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.305806 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.305790    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.306216 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.306192    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.306307 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.306252    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.306498 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.306484    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.306720 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.306699    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-web-config\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.339304 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.339281    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jtt\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-kube-api-access-k2jtt\") pod \"alertmanager-main-0\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.392130 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.392110    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:52:49.564494 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.564467    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:52:49.566434 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:49.566403    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6247555_988e_455c_b8dd_de06e339d0d7.slice/crio-f5b844534a8a1e83b1310265e2ae50519f4375e199f7ff79b34e2e9c2a95092b WatchSource:0}: Error finding container f5b844534a8a1e83b1310265e2ae50519f4375e199f7ff79b34e2e9c2a95092b: Status 404 returned error can't find the container with id f5b844534a8a1e83b1310265e2ae50519f4375e199f7ff79b34e2e9c2a95092b
May 20 13:52:49.709480 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.709452    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"]
May 20 13:52:49.712267 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.712240    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.717435 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.717406    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 20 13:52:49.717591 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.717406    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 20 13:52:49.717768 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.717740    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 20 13:52:49.717768 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.717751    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hmxcz\""
May 20 13:52:49.717915 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.717751    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 20 13:52:49.718212 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.718193    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-arospu1tkjide\""
May 20 13:52:49.718305 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.718233    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 20 13:52:49.734065 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.734027    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"]
May 20 13:52:49.806276 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.806253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.806429 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.806307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.806429 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.806371    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-tls\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.806549 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.806431    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.806549 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.806451    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.806549 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.806476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-grpc-tls\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.806549 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.806500    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854f5\" (UniqueName: \"kubernetes.io/projected/607041de-41d1-4277-b2c4-e6b7f354c9a6-kube-api-access-854f5\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.806549 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.806542    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/607041de-41d1-4277-b2c4-e6b7f354c9a6-metrics-client-ca\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.908326 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.907302    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/607041de-41d1-4277-b2c4-e6b7f354c9a6-metrics-client-ca\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.908326 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.907380    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.908326 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.907428    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.908326 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.907470    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-tls\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.908326 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.907510    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.908326 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.907533    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.908326 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.907553    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-grpc-tls\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.908326 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.907576    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-854f5\" (UniqueName: \"kubernetes.io/projected/607041de-41d1-4277-b2c4-e6b7f354c9a6-kube-api-access-854f5\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.908788 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.908551    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/607041de-41d1-4277-b2c4-e6b7f354c9a6-metrics-client-ca\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.911256 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.911228    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.911369 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.911285    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.911369 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.911313    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-grpc-tls\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.911648 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.911628    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.911726 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.911666    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-tls\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.911833 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.911816    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/607041de-41d1-4277-b2c4-e6b7f354c9a6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.917174 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.917156    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854f5\" (UniqueName: \"kubernetes.io/projected/607041de-41d1-4277-b2c4-e6b7f354c9a6-kube-api-access-854f5\") pod \"thanos-querier-6dc49d9f-5kmkc\" (UID: \"607041de-41d1-4277-b2c4-e6b7f354c9a6\") " pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:49.949734 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.949709    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49" event={"ID":"bb528ea5-f9cc-4ff8-91d2-49c5452bf8be","Type":"ContainerStarted","Data":"0813c0dcaf369d5086d39ba505be2592e62c5db0ea8e8127ba416bc570dc8182"}
May 20 13:52:49.951028 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.951005    2557 generic.go:358] "Generic (PLEG): container finished" podID="9bac2ac0-f861-498f-92f6-049402b8f6d6" containerID="fa0e35dcf39096dc1bca78d1fd49d1d8c1a70239c8115966014ab4cb197da1ae" exitCode=0
May 20 13:52:49.951134 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.951085    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sx46z" event={"ID":"9bac2ac0-f861-498f-92f6-049402b8f6d6","Type":"ContainerDied","Data":"fa0e35dcf39096dc1bca78d1fd49d1d8c1a70239c8115966014ab4cb197da1ae"}
May 20 13:52:49.952174 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.952152    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerStarted","Data":"f5b844534a8a1e83b1310265e2ae50519f4375e199f7ff79b34e2e9c2a95092b"}
May 20 13:52:49.983046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:49.982964    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-6sb49" podStartSLOduration=1.97992054 podStartE2EDuration="2.982949825s" podCreationTimestamp="2026-05-20 13:52:47 +0000 UTC" firstStartedPulling="2026-05-20 13:52:48.029783581 +0000 UTC m=+148.167337777" lastFinishedPulling="2026-05-20 13:52:49.032812853 +0000 UTC m=+149.170367062" observedRunningTime="2026-05-20 13:52:49.982747485 +0000 UTC m=+150.120301705" watchObservedRunningTime="2026-05-20 13:52:49.982949825 +0000 UTC m=+150.120504044"
May 20 13:52:50.024005 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.023941    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:50.201491 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.201395    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"]
May 20 13:52:50.205307 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:50.205276    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607041de_41d1_4277_b2c4_e6b7f354c9a6.slice/crio-75aad8ff5a0502595a43bd8f99696ff5ea5b2759fc1337029053f7c1b52886d9 WatchSource:0}: Error finding container 75aad8ff5a0502595a43bd8f99696ff5ea5b2759fc1337029053f7c1b52886d9: Status 404 returned error can't find the container with id 75aad8ff5a0502595a43bd8f99696ff5ea5b2759fc1337029053f7c1b52886d9
May 20 13:52:50.956074 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.956001    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc" event={"ID":"607041de-41d1-4277-b2c4-e6b7f354c9a6","Type":"ContainerStarted","Data":"75aad8ff5a0502595a43bd8f99696ff5ea5b2759fc1337029053f7c1b52886d9"}
May 20 13:52:50.958210 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.958157    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sx46z" event={"ID":"9bac2ac0-f861-498f-92f6-049402b8f6d6","Type":"ContainerStarted","Data":"490d5cacb54e26fa88b26caebd95e620b1cd1665441a4a18959183a2f77b02b0"}
May 20 13:52:50.958210 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.958187    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sx46z" event={"ID":"9bac2ac0-f861-498f-92f6-049402b8f6d6","Type":"ContainerStarted","Data":"c105153745e25c78c4b05887a2975a966604cd4c10fa3bfbe48a5a814479bf90"}
May 20 13:52:50.960273 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.960245    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf" event={"ID":"011b9ea8-4788-46a9-9cb6-d5c6f20b843d","Type":"ContainerStarted","Data":"63302c6a2b4062601dcb87e456d552f0489f3319f45f035c1466d37d44174ad5"}
May 20 13:52:50.960382 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.960281    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf" event={"ID":"011b9ea8-4788-46a9-9cb6-d5c6f20b843d","Type":"ContainerStarted","Data":"f76ca3bd2577400b16752333164439b919a70e17d9b4e5cc830ec5a92aac5c6b"}
May 20 13:52:50.960382 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.960294    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf" event={"ID":"011b9ea8-4788-46a9-9cb6-d5c6f20b843d","Type":"ContainerStarted","Data":"0c3f3ccc3bf063f9fb96fc62920efb92543fc1f47fced1089caeb9a37ca20668"}
May 20 13:52:50.961721 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.961693    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6247555-988e-455c-b8dd-de06e339d0d7" containerID="869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0" exitCode=0
May 20 13:52:50.961843 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:50.961775    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerDied","Data":"869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0"}
May 20 13:52:51.040886 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:51.040824    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sx46z" podStartSLOduration=3.181633268 podStartE2EDuration="4.040810047s" podCreationTimestamp="2026-05-20 13:52:47 +0000 UTC" firstStartedPulling="2026-05-20 13:52:47.874599194 +0000 UTC m=+148.012153390" lastFinishedPulling="2026-05-20 13:52:48.733775973 +0000 UTC m=+148.871330169" observedRunningTime="2026-05-20 13:52:50.988231731 +0000 UTC m=+151.125785948" watchObservedRunningTime="2026-05-20 13:52:51.040810047 +0000 UTC m=+151.178364267"
May 20 13:52:51.072574 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:51.072527    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-rvzhf" podStartSLOduration=2.845485384 podStartE2EDuration="4.072515725s" podCreationTimestamp="2026-05-20 13:52:47 +0000 UTC" firstStartedPulling="2026-05-20 13:52:48.710131794 +0000 UTC m=+148.847686001" lastFinishedPulling="2026-05-20 13:52:49.937162143 +0000 UTC m=+150.074716342" observedRunningTime="2026-05-20 13:52:51.072457551 +0000 UTC m=+151.210011769" watchObservedRunningTime="2026-05-20 13:52:51.072515725 +0000 UTC m=+151.210069942"
May 20 13:52:52.968733 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:52.968701    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc" event={"ID":"607041de-41d1-4277-b2c4-e6b7f354c9a6","Type":"ContainerStarted","Data":"37ffe5487c86db3809ce318f7854ef44f8690d527cad6b30bd1c505c3165688c"}
May 20 13:52:52.969046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:52.968744    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc" event={"ID":"607041de-41d1-4277-b2c4-e6b7f354c9a6","Type":"ContainerStarted","Data":"abef3bd04b1396f7f4514701e04f9823bbf6db3d4795b73f0089a0d6fed05cec"}
May 20 13:52:52.969046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:52.968756    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc" event={"ID":"607041de-41d1-4277-b2c4-e6b7f354c9a6","Type":"ContainerStarted","Data":"d9b444e75d3bce5fab29c80d6c93adf3c00a7d60ee1075c83eb9e15dfac0df34"}
May 20 13:52:52.971269 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:52.971249    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerStarted","Data":"63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6"}
May 20 13:52:52.971348 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:52.971275    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerStarted","Data":"b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3"}
May 20 13:52:52.971348 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:52.971284    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerStarted","Data":"393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0"}
May 20 13:52:52.971348 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:52.971292    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerStarted","Data":"8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3"}
May 20 13:52:52.971348 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:52.971301    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerStarted","Data":"1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9"}
May 20 13:52:53.977335 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:53.977255    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerStarted","Data":"3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0"}
May 20 13:52:53.979751 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:53.979727    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc" event={"ID":"607041de-41d1-4277-b2c4-e6b7f354c9a6","Type":"ContainerStarted","Data":"95f9b0e674e72e45cee826804978f76759465303f58f9acac85f47fd5a22c327"}
May 20 13:52:53.979751 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:53.979756    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc" event={"ID":"607041de-41d1-4277-b2c4-e6b7f354c9a6","Type":"ContainerStarted","Data":"28665a21d404f584fe06f989ee64218cd874a3cfe20037abdf573c531d5924d8"}
May 20 13:52:53.979885 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:53.979767    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc" event={"ID":"607041de-41d1-4277-b2c4-e6b7f354c9a6","Type":"ContainerStarted","Data":"5538779868b61aee277347430bf67de384ae046ccc1aae609a0c3dc5be2b570c"}
May 20 13:52:53.979917 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:53.979893    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:52:54.030788 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.030754    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:52:54.031366 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.031324    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=0.922924313 podStartE2EDuration="5.031309925s" podCreationTimestamp="2026-05-20 13:52:49 +0000 UTC" firstStartedPulling="2026-05-20 13:52:49.568712875 +0000 UTC m=+149.706267071" lastFinishedPulling="2026-05-20 13:52:53.677098479 +0000 UTC m=+153.814652683" observedRunningTime="2026-05-20 13:52:54.027160326 +0000 UTC m=+154.164714543" watchObservedRunningTime="2026-05-20 13:52:54.031309925 +0000 UTC m=+154.168864143"
May 20 13:52:54.035087 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.035070    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.037977 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.037958    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 20 13:52:54.038089 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.037981    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 20 13:52:54.038089 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.038054    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 20 13:52:54.038291 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.038274    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-frs7k\""
May 20 13:52:54.038372 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.038356    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5419umi8scg9s\""
May 20 13:52:54.038440 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.038428    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 20 13:52:54.039266 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.039133    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 20 13:52:54.039266 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.039146    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 20 13:52:54.039266 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.039160    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 20 13:52:54.039266 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.039176    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 20 13:52:54.039266 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.039192    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 20 13:52:54.039266 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.039208    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 20 13:52:54.039266 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.039133    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 20 13:52:54.043222 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.043203    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 20 13:52:54.045317 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.045298    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 20 13:52:54.059221 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.059201    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:52:54.070440 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.070398    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc" podStartSLOduration=1.599323525 podStartE2EDuration="5.070387181s" podCreationTimestamp="2026-05-20 13:52:49 +0000 UTC" firstStartedPulling="2026-05-20 13:52:50.207747486 +0000 UTC m=+150.345301683" lastFinishedPulling="2026-05-20 13:52:53.678811133 +0000 UTC m=+153.816365339" observedRunningTime="2026-05-20 13:52:54.069052521 +0000 UTC m=+154.206606744" watchObservedRunningTime="2026-05-20 13:52:54.070387181 +0000 UTC m=+154.207941398"
May 20 13:52:54.149019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.148996    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149102 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149033    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149170 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149126    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149256 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149232    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149314 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149296    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149366 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149326    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149366 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149366 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149361    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149491 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149548 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149517    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149548 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149776 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149729    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149776 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149764    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.149949 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149904    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.150004 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.149974    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.150050 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.150001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.150085 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.150077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvrn\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-kube-api-access-gmvrn\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.150128 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.150106    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251358 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251300    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251358 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251338    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251358 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251357    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251495 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251471    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251526 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251504    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251562 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251529    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251607 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251564    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251607 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251592    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251712 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251620    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251712 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251658    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251712 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251682    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251886 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251725    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251886 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251774    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251886 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251806    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.251886 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251842    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvrn\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-kube-api-access-gmvrn\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.252091 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251905    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.252091 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251946    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.252091 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.251971    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.252312 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.252286    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.254401 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.254378    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.254650 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.254622    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.254758 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.254737    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.254973 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.254834    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.255166 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.255141    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.255166 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.255157    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.255365 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.255344    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.255585 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.255554    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.255660 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.255585    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.255725 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.255674    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.256028 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.255998    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.256191 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.256173    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.256595 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.256574    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.257164 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.257146    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.257719 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.257697    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.257820 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.257804    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.264102 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.264084    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvrn\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-kube-api-access-gmvrn\") pod \"prometheus-k8s-0\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.344155 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.344132    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:54.475366 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.475335    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:52:54.478039 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:52:54.478016    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc0dc8f_3d23_4563_a8b6_5ca482a750bd.slice/crio-ea68b5461a08b3a4874e222bf817be4e546645886bbb3abcc9147b59bccd059e WatchSource:0}: Error finding container ea68b5461a08b3a4874e222bf817be4e546645886bbb3abcc9147b59bccd059e: Status 404 returned error can't find the container with id ea68b5461a08b3a4874e222bf817be4e546645886bbb3abcc9147b59bccd059e
May 20 13:52:54.985190 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.985159    2557 generic.go:358] "Generic (PLEG): container finished" podID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerID="c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611" exitCode=0
May 20 13:52:54.985549 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.985247    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerDied","Data":"c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611"}
May 20 13:52:54.985549 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:54.985289    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerStarted","Data":"ea68b5461a08b3a4874e222bf817be4e546645886bbb3abcc9147b59bccd059e"}
May 20 13:52:56.283994 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:56.283950    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8vcdb" podUID="ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75"
May 20 13:52:56.338248 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:52:56.338212    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-4qp54" podUID="0cfdbcdd-0be7-42e1-91eb-9892ee0ea925"
May 20 13:52:56.992902 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:56.992870    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vcdb"
May 20 13:52:57.401464 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:57.401437    2557 scope.go:117] "RemoveContainer" containerID="ce6194e3f1dcc178003421d3860724976c53bbe35907c6a3c631d621d9535930"
May 20 13:52:57.998873 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:57.998828    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerStarted","Data":"6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66"}
May 20 13:52:57.998991 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:57.998886    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerStarted","Data":"44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8"}
May 20 13:52:57.998991 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:57.998901    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerStarted","Data":"321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3"}
May 20 13:52:57.998991 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:57.998918    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerStarted","Data":"08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0"}
May 20 13:52:57.998991 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:57.998931    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerStarted","Data":"b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21"}
May 20 13:52:57.998991 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:57.998944    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerStarted","Data":"f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9"}
May 20 13:52:58.000418 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:58.000400    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 13:52:58.000504 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:58.000442    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" event={"ID":"9ed23ead-6ce7-4b98-a8f3-042a5d72979c","Type":"ContainerStarted","Data":"97a3f85aaa3829d4fa6d8997d91f1b2597ce34d966de3158cff46b6fbb28e9cd"}
May 20 13:52:58.000704 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:58.000682    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:58.028095 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:58.028071    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-vgn62"
May 20 13:52:58.030976 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:58.030939    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.496481221 podStartE2EDuration="5.03092591s" podCreationTimestamp="2026-05-20 13:52:53 +0000 UTC" firstStartedPulling="2026-05-20 13:52:54.986431223 +0000 UTC m=+155.123985419" lastFinishedPulling="2026-05-20 13:52:57.520875905 +0000 UTC m=+157.658430108" observedRunningTime="2026-05-20 13:52:58.029136471 +0000 UTC m=+158.166690726" watchObservedRunningTime="2026-05-20 13:52:58.03092591 +0000 UTC m=+158.168480128"
May 20 13:52:58.058795 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:58.058761    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-vgn62" podStartSLOduration=53.100144265 podStartE2EDuration="55.058749253s" podCreationTimestamp="2026-05-20 13:52:03 +0000 UTC" firstStartedPulling="2026-05-20 13:52:04.113606194 +0000 UTC m=+104.251160404" lastFinishedPulling="2026-05-20 13:52:06.072211193 +0000 UTC m=+106.209765392" observedRunningTime="2026-05-20 13:52:58.056965882 +0000 UTC m=+158.194520100" watchObservedRunningTime="2026-05-20 13:52:58.058749253 +0000 UTC m=+158.196303470"
May 20 13:52:59.344260 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:59.344221    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:52:59.992875 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:52:59.992834    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6dc49d9f-5kmkc"
May 20 13:53:01.414551 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:01.414445    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:53:01.414551 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:01.414515    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:53:01.417136 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:01.417105    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75-metrics-tls\") pod \"dns-default-8vcdb\" (UID: \"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75\") " pod="openshift-dns/dns-default-8vcdb"
May 20 13:53:01.417298 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:01.417276    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cfdbcdd-0be7-42e1-91eb-9892ee0ea925-cert\") pod \"ingress-canary-4qp54\" (UID: \"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925\") " pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:53:01.496222 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:01.496199    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wmcgw\""
May 20 13:53:01.503846 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:01.503826    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vcdb"
May 20 13:53:01.631936 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:01.631905    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8vcdb"]
May 20 13:53:01.634183 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:53:01.634146    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1c4d6d_dcf4_4d04_8e49_c846e28d1f75.slice/crio-480f9dc1cf48718c065623ae8d663d7d53ea5bdc007e087bbcb43d45f605c88b WatchSource:0}: Error finding container 480f9dc1cf48718c065623ae8d663d7d53ea5bdc007e087bbcb43d45f605c88b: Status 404 returned error can't find the container with id 480f9dc1cf48718c065623ae8d663d7d53ea5bdc007e087bbcb43d45f605c88b
May 20 13:53:02.015149 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:02.015108    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8vcdb" event={"ID":"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75","Type":"ContainerStarted","Data":"480f9dc1cf48718c065623ae8d663d7d53ea5bdc007e087bbcb43d45f605c88b"}
May 20 13:53:04.022672 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:04.022637    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8vcdb" event={"ID":"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75","Type":"ContainerStarted","Data":"6452c526617df84238ebefac4504fe10a2b08bdf8da9dd817bfcf86aa5c10973"}
May 20 13:53:04.022672 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:04.022674    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8vcdb" event={"ID":"ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75","Type":"ContainerStarted","Data":"656d56e83988f262aabd279f1c8cce6b1780f2464a28ad27f6d97ada57a65e81"}
May 20 13:53:04.023107 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:04.022753    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8vcdb"
May 20 13:53:04.048334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:04.048280    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-8vcdb" podStartSLOduration=129.464264141 podStartE2EDuration="2m11.0482673s" podCreationTimestamp="2026-05-20 13:50:53 +0000 UTC" firstStartedPulling="2026-05-20 13:53:01.635966252 +0000 UTC m=+161.773520448" lastFinishedPulling="2026-05-20 13:53:03.219969411 +0000 UTC m=+163.357523607" observedRunningTime="2026-05-20 13:53:04.048077354 +0000 UTC m=+164.185631572" watchObservedRunningTime="2026-05-20 13:53:04.0482673 +0000 UTC m=+164.185821613"
May 20 13:53:07.401234 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:07.401195    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:53:07.404211 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:07.404189    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8tpd8\""
May 20 13:53:07.411676 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:07.411649    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4qp54"
May 20 13:53:07.535244 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:07.535130    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4qp54"]
May 20 13:53:07.537889 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:53:07.537849    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfdbcdd_0be7_42e1_91eb_9892ee0ea925.slice/crio-a4c4bc129a3bc714527a48e5bca6fe97cb2c19c66bc6a046c959e5c13c049b8e WatchSource:0}: Error finding container a4c4bc129a3bc714527a48e5bca6fe97cb2c19c66bc6a046c959e5c13c049b8e: Status 404 returned error can't find the container with id a4c4bc129a3bc714527a48e5bca6fe97cb2c19c66bc6a046c959e5c13c049b8e
May 20 13:53:08.040899 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:08.040818    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4qp54" event={"ID":"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925","Type":"ContainerStarted","Data":"a4c4bc129a3bc714527a48e5bca6fe97cb2c19c66bc6a046c959e5c13c049b8e"}
May 20 13:53:10.049506 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:10.049465    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4qp54" event={"ID":"0cfdbcdd-0be7-42e1-91eb-9892ee0ea925","Type":"ContainerStarted","Data":"40cab1ddd6a2f51aee964849b46621159e1419edc41f9586bf4f91d96715e43a"}
May 20 13:53:10.068237 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:10.068182    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4qp54" podStartSLOduration=135.556072091 podStartE2EDuration="2m17.068139262s" podCreationTimestamp="2026-05-20 13:50:53 +0000 UTC" firstStartedPulling="2026-05-20 13:53:07.539719465 +0000 UTC m=+167.677273677" lastFinishedPulling="2026-05-20 13:53:09.051786644 +0000 UTC m=+169.189340848" observedRunningTime="2026-05-20 13:53:10.067332197 +0000 UTC m=+170.204886416" watchObservedRunningTime="2026-05-20 13:53:10.068139262 +0000 UTC m=+170.205693479"
May 20 13:53:14.028462 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:14.028428    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8vcdb"
May 20 13:53:37.138545 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:37.138508    2557 generic.go:358] "Generic (PLEG): container finished" podID="42b21860-0b33-45ef-b866-d55adf52bbc8" containerID="4ef3b1902b4b01cd48e052cd4172ed4095f597a397a17fde88c0b68f3153f6c8" exitCode=0
May 20 13:53:37.138943 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:37.138582    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc" event={"ID":"42b21860-0b33-45ef-b866-d55adf52bbc8","Type":"ContainerDied","Data":"4ef3b1902b4b01cd48e052cd4172ed4095f597a397a17fde88c0b68f3153f6c8"}
May 20 13:53:37.138943 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:37.138941    2557 scope.go:117] "RemoveContainer" containerID="4ef3b1902b4b01cd48e052cd4172ed4095f597a397a17fde88c0b68f3153f6c8"
May 20 13:53:38.143776 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:38.143741    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-s25dc" event={"ID":"42b21860-0b33-45ef-b866-d55adf52bbc8","Type":"ContainerStarted","Data":"1028d7e056e9279d4f43319dfb307f04de10446674733ec2b2beefbf767360f1"}
May 20 13:53:54.344458 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:54.344421    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:54.360820 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:54.360795    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:53:55.212932 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:53:55.212904    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:08.181611 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:08.181568    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:08.182174 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:08.182093    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="alertmanager" containerID="cri-o://1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9" gracePeriod=120
May 20 13:54:08.182174 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:08.182152    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy-metric" containerID="cri-o://63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6" gracePeriod=120
May 20 13:54:08.182174 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:08.182164    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy" containerID="cri-o://b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3" gracePeriod=120
May 20 13:54:08.182484 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:08.182152    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy-web" containerID="cri-o://393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0" gracePeriod=120
May 20 13:54:08.182484 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:08.182209    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="prom-label-proxy" containerID="cri-o://3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0" gracePeriod=120
May 20 13:54:08.182484 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:08.182258    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="config-reloader" containerID="cri-o://8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3" gracePeriod=120
May 20 13:54:09.248739 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.248710    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6247555-988e-455c-b8dd-de06e339d0d7" containerID="3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0" exitCode=0
May 20 13:54:09.248739 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.248733    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6247555-988e-455c-b8dd-de06e339d0d7" containerID="b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3" exitCode=0
May 20 13:54:09.248739 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.248740    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6247555-988e-455c-b8dd-de06e339d0d7" containerID="8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3" exitCode=0
May 20 13:54:09.249129 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.248746    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6247555-988e-455c-b8dd-de06e339d0d7" containerID="1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9" exitCode=0
May 20 13:54:09.249129 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.248783    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerDied","Data":"3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0"}
May 20 13:54:09.249129 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.248819    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerDied","Data":"b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3"}
May 20 13:54:09.249129 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.248831    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerDied","Data":"8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3"}
May 20 13:54:09.249129 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.248840    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerDied","Data":"1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9"}
May 20 13:54:09.427505 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.427483    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:09.525699 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.525638    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-config-volume\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.525699 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.525671    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-web-config\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.525908 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.525708    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-web\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.525908 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.525819    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-tls-assets\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.525908 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.525872    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.526081 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.525929    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2jtt\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-kube-api-access-k2jtt\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.526081 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.525975    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-main-tls\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.526081 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.526018    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.526081 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.526049    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-main-db\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.526279 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.526085    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-cluster-tls-config\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.526279 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.526143    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-metrics-client-ca\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.526279 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.526180    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-trusted-ca-bundle\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.526279 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.526208    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-config-out\") pod \"e6247555-988e-455c-b8dd-de06e339d0d7\" (UID: \"e6247555-988e-455c-b8dd-de06e339d0d7\") "
May 20 13:54:09.527198 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.527141    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:09.527198 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.527173    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:54:09.527549 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.527513    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:09.528512 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.528488    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:09.529760 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.529723    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:09.530079 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.529984    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:09.530079 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.529993    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:09.530079 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.530041    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-config-volume" (OuterVolumeSpecName: "config-volume") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:09.530079 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.530057    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-config-out" (OuterVolumeSpecName: "config-out") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:54:09.530274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.530121    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-kube-api-access-k2jtt" (OuterVolumeSpecName: "kube-api-access-k2jtt") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "kube-api-access-k2jtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:09.530274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.530120    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:09.533049 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.533030    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:09.539372 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.539353    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-web-config" (OuterVolumeSpecName: "web-config") pod "e6247555-988e-455c-b8dd-de06e339d0d7" (UID: "e6247555-988e-455c-b8dd-de06e339d0d7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:09.626707 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626689    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-web-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626707 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626708    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626718    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-tls-assets\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626727    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626735    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2jtt\" (UniqueName: \"kubernetes.io/projected/e6247555-988e-455c-b8dd-de06e339d0d7-kube-api-access-k2jtt\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626744    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-main-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626754    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626763    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-main-db\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626771    2557 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-cluster-tls-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626779    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-metrics-client-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626787    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6247555-988e-455c-b8dd-de06e339d0d7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626796    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6247555-988e-455c-b8dd-de06e339d0d7-config-out\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:09.626815 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:09.626803    2557 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e6247555-988e-455c-b8dd-de06e339d0d7-config-volume\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:10.255191 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.255158    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6247555-988e-455c-b8dd-de06e339d0d7" containerID="63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6" exitCode=0
May 20 13:54:10.255191 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.255187    2557 generic.go:358] "Generic (PLEG): container finished" podID="e6247555-988e-455c-b8dd-de06e339d0d7" containerID="393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0" exitCode=0
May 20 13:54:10.255541 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.255237    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerDied","Data":"63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6"}
May 20 13:54:10.255541 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.255281    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerDied","Data":"393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0"}
May 20 13:54:10.255541 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.255284    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.255541 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.255293    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e6247555-988e-455c-b8dd-de06e339d0d7","Type":"ContainerDied","Data":"f5b844534a8a1e83b1310265e2ae50519f4375e199f7ff79b34e2e9c2a95092b"}
May 20 13:54:10.255541 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.255310    2557 scope.go:117] "RemoveContainer" containerID="3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0"
May 20 13:54:10.263667 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.263561    2557 scope.go:117] "RemoveContainer" containerID="63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6"
May 20 13:54:10.270403 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.270387    2557 scope.go:117] "RemoveContainer" containerID="b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3"
May 20 13:54:10.276524 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.276508    2557 scope.go:117] "RemoveContainer" containerID="393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0"
May 20 13:54:10.282942 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.282924    2557 scope.go:117] "RemoveContainer" containerID="8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3"
May 20 13:54:10.290063 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.290038    2557 scope.go:117] "RemoveContainer" containerID="1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9"
May 20 13:54:10.291017 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.290995    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:10.294154 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.294127    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:10.297885 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.297731    2557 scope.go:117] "RemoveContainer" containerID="869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0"
May 20 13:54:10.303899 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.303882    2557 scope.go:117] "RemoveContainer" containerID="3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0"
May 20 13:54:10.304143 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:10.304124    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0\": container with ID starting with 3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0 not found: ID does not exist" containerID="3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0"
May 20 13:54:10.304218 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.304153    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0"} err="failed to get container status \"3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0\": rpc error: code = NotFound desc = could not find container \"3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0\": container with ID starting with 3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0 not found: ID does not exist"
May 20 13:54:10.304218 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.304195    2557 scope.go:117] "RemoveContainer" containerID="63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6"
May 20 13:54:10.304448 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:10.304431    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6\": container with ID starting with 63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6 not found: ID does not exist" containerID="63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6"
May 20 13:54:10.304485 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.304453    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6"} err="failed to get container status \"63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6\": rpc error: code = NotFound desc = could not find container \"63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6\": container with ID starting with 63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6 not found: ID does not exist"
May 20 13:54:10.304485 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.304469    2557 scope.go:117] "RemoveContainer" containerID="b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3"
May 20 13:54:10.304679 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:10.304664    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3\": container with ID starting with b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3 not found: ID does not exist" containerID="b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3"
May 20 13:54:10.304724 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.304683    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3"} err="failed to get container status \"b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3\": rpc error: code = NotFound desc = could not find container \"b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3\": container with ID starting with b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3 not found: ID does not exist"
May 20 13:54:10.304724 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.304696    2557 scope.go:117] "RemoveContainer" containerID="393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0"
May 20 13:54:10.304902 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:10.304885    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0\": container with ID starting with 393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0 not found: ID does not exist" containerID="393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0"
May 20 13:54:10.304971 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.304911    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0"} err="failed to get container status \"393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0\": rpc error: code = NotFound desc = could not find container \"393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0\": container with ID starting with 393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0 not found: ID does not exist"
May 20 13:54:10.304971 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.304931    2557 scope.go:117] "RemoveContainer" containerID="8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3"
May 20 13:54:10.305178 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:10.305162    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3\": container with ID starting with 8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3 not found: ID does not exist" containerID="8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3"
May 20 13:54:10.305265 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.305184    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3"} err="failed to get container status \"8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3\": rpc error: code = NotFound desc = could not find container \"8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3\": container with ID starting with 8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3 not found: ID does not exist"
May 20 13:54:10.305265 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.305198    2557 scope.go:117] "RemoveContainer" containerID="1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9"
May 20 13:54:10.305399 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:10.305386    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9\": container with ID starting with 1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9 not found: ID does not exist" containerID="1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9"
May 20 13:54:10.305438 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.305402    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9"} err="failed to get container status \"1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9\": rpc error: code = NotFound desc = could not find container \"1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9\": container with ID starting with 1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9 not found: ID does not exist"
May 20 13:54:10.305438 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.305413    2557 scope.go:117] "RemoveContainer" containerID="869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0"
May 20 13:54:10.305579 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:10.305563    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0\": container with ID starting with 869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0 not found: ID does not exist" containerID="869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0"
May 20 13:54:10.305636 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.305587    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0"} err="failed to get container status \"869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0\": rpc error: code = NotFound desc = could not find container \"869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0\": container with ID starting with 869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0 not found: ID does not exist"
May 20 13:54:10.305636 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.305608    2557 scope.go:117] "RemoveContainer" containerID="3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0"
May 20 13:54:10.305800 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.305785    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0"} err="failed to get container status \"3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0\": rpc error: code = NotFound desc = could not find container \"3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0\": container with ID starting with 3a58b02f03467e2c9fcaec6db6b05c2881741995b63217546e844885889100d0 not found: ID does not exist"
May 20 13:54:10.305878 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.305802    2557 scope.go:117] "RemoveContainer" containerID="63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6"
May 20 13:54:10.306005 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.305986    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6"} err="failed to get container status \"63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6\": rpc error: code = NotFound desc = could not find container \"63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6\": container with ID starting with 63067ebabc3bdfbec194ca0f6980894bab6bab77d5f56012453fcca0024746f6 not found: ID does not exist"
May 20 13:54:10.306058 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.306006    2557 scope.go:117] "RemoveContainer" containerID="b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3"
May 20 13:54:10.306236 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.306218    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3"} err="failed to get container status \"b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3\": rpc error: code = NotFound desc = could not find container \"b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3\": container with ID starting with b811c8f40ecd584adff064bcace8203ab6ad3f39fc230aab6aa51cf27fd874c3 not found: ID does not exist"
May 20 13:54:10.306279 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.306236    2557 scope.go:117] "RemoveContainer" containerID="393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0"
May 20 13:54:10.306441 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.306424    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0"} err="failed to get container status \"393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0\": rpc error: code = NotFound desc = could not find container \"393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0\": container with ID starting with 393c57cf37f4df5deab8662a07a368917f12a2650860d898800adbd4bd0a5fa0 not found: ID does not exist"
May 20 13:54:10.306504 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.306445    2557 scope.go:117] "RemoveContainer" containerID="8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3"
May 20 13:54:10.306679 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.306660    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3"} err="failed to get container status \"8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3\": rpc error: code = NotFound desc = could not find container \"8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3\": container with ID starting with 8f70e92e40314599d1f98f9c2f594583dcdd3d0464972677ef5c5ef5f607adb3 not found: ID does not exist"
May 20 13:54:10.306722 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.306681    2557 scope.go:117] "RemoveContainer" containerID="1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9"
May 20 13:54:10.306916 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.306879    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9"} err="failed to get container status \"1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9\": rpc error: code = NotFound desc = could not find container \"1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9\": container with ID starting with 1bf65d5d3ededbc976243cb84b87823a358a27f4790f1e17486280862970a7f9 not found: ID does not exist"
May 20 13:54:10.306916 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.306907    2557 scope.go:117] "RemoveContainer" containerID="869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0"
May 20 13:54:10.307148 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.307131    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0"} err="failed to get container status \"869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0\": rpc error: code = NotFound desc = could not find container \"869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0\": container with ID starting with 869c905ed385012098c358ce9d096cdd1ab853e11dd4fe90192c3b8b46dc8bb0 not found: ID does not exist"
May 20 13:54:10.322633 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.322611    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:10.322960 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.322947    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="alertmanager"
May 20 13:54:10.323002 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.322962    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="alertmanager"
May 20 13:54:10.323002 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.322972    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="prom-label-proxy"
May 20 13:54:10.323002 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.322977    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="prom-label-proxy"
May 20 13:54:10.323002 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.322986    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy"
May 20 13:54:10.323002 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.322991    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy"
May 20 13:54:10.323002 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323000    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy-metric"
May 20 13:54:10.323002 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323005    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy-metric"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323012    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy-web"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323017    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy-web"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323027    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="init-config-reloader"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323032    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="init-config-reloader"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323044    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="config-reloader"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323052    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="config-reloader"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323114    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="alertmanager"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323121    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323128    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="config-reloader"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323135    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="prom-label-proxy"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323142    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy-web"
May 20 13:54:10.323228 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.323151    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" containerName="kube-rbac-proxy-metric"
May 20 13:54:10.327107 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.327091    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.330090 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.329963    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 20 13:54:10.330090 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.330039    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 20 13:54:10.330090 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.330056    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 20 13:54:10.330090 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.330056    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 20 13:54:10.330090 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.330092    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7q8hw\""
May 20 13:54:10.330364 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.330044    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 20 13:54:10.330364 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.330060    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 20 13:54:10.330690 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.330678    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 20 13:54:10.331385 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.331371    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 20 13:54:10.336619 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.336603    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 20 13:54:10.341835 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.341814    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:10.409271 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.409243    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6247555-988e-455c-b8dd-de06e339d0d7" path="/var/lib/kubelet/pods/e6247555-988e-455c-b8dd-de06e339d0d7/volumes"
May 20 13:54:10.430586 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430566    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sssqx\" (UniqueName: \"kubernetes.io/projected/71936781-21e0-4444-9bc4-c23d7840ae67-kube-api-access-sssqx\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430655 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430596    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-web-config\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430655 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430623    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71936781-21e0-4444-9bc4-c23d7840ae67-tls-assets\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430723 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430667    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430723 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71936781-21e0-4444-9bc4-c23d7840ae67-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430782 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430734    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430782 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/71936781-21e0-4444-9bc4-c23d7840ae67-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430842 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430788    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430842 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430822    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430916 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430884    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-config-volume\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430916 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430901    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430979 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430917    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71936781-21e0-4444-9bc4-c23d7840ae67-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.430979 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.430942    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71936781-21e0-4444-9bc4-c23d7840ae67-config-out\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.531878 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.531801    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71936781-21e0-4444-9bc4-c23d7840ae67-tls-assets\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.531878 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.531838    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532015 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.531889    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71936781-21e0-4444-9bc4-c23d7840ae67-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532015 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.531909    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532015 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.531928    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/71936781-21e0-4444-9bc4-c23d7840ae67-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532015 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.531960    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532015 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.531983    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532015 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.532007    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-config-volume\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532319 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.532023    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532319 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.532043    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71936781-21e0-4444-9bc4-c23d7840ae67-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532319 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.532068    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71936781-21e0-4444-9bc4-c23d7840ae67-config-out\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532319 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.532103    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sssqx\" (UniqueName: \"kubernetes.io/projected/71936781-21e0-4444-9bc4-c23d7840ae67-kube-api-access-sssqx\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532319 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.532126    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-web-config\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.532558 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.532326    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/71936781-21e0-4444-9bc4-c23d7840ae67-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.533364 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.533336    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71936781-21e0-4444-9bc4-c23d7840ae67-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.534806 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.534783    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/71936781-21e0-4444-9bc4-c23d7840ae67-config-out\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.534916 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.534810    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.535058 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.535025    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/71936781-21e0-4444-9bc4-c23d7840ae67-tls-assets\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.535159 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.535034    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.535412 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.535391    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71936781-21e0-4444-9bc4-c23d7840ae67-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.535497 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.535418    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.535497 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.535479    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.535601 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.535507    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-web-config\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.535829 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.535806    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.537070 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.537053    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/71936781-21e0-4444-9bc4-c23d7840ae67-config-volume\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.550849 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.550831    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sssqx\" (UniqueName: \"kubernetes.io/projected/71936781-21e0-4444-9bc4-c23d7840ae67-kube-api-access-sssqx\") pod \"alertmanager-main-0\" (UID: \"71936781-21e0-4444-9bc4-c23d7840ae67\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.635886 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.635848    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:10.763442 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:10.763386    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:10.766905 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:54:10.766879    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71936781_21e0_4444_9bc4_c23d7840ae67.slice/crio-182bcce83d4ed96d3526a4c0365531af20d9a08e6d7fbf020f528cfda2bbf42e WatchSource:0}: Error finding container 182bcce83d4ed96d3526a4c0365531af20d9a08e6d7fbf020f528cfda2bbf42e: Status 404 returned error can't find the container with id 182bcce83d4ed96d3526a4c0365531af20d9a08e6d7fbf020f528cfda2bbf42e
May 20 13:54:11.260577 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:11.260546    2557 generic.go:358] "Generic (PLEG): container finished" podID="71936781-21e0-4444-9bc4-c23d7840ae67" containerID="dc40de47216a87c67d88f2c90c01a3b3fb890d59b1f86f5cbebcfcb3abd2b017" exitCode=0
May 20 13:54:11.260926 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:11.260588    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71936781-21e0-4444-9bc4-c23d7840ae67","Type":"ContainerDied","Data":"dc40de47216a87c67d88f2c90c01a3b3fb890d59b1f86f5cbebcfcb3abd2b017"}
May 20 13:54:11.260926 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:11.260611    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71936781-21e0-4444-9bc4-c23d7840ae67","Type":"ContainerStarted","Data":"182bcce83d4ed96d3526a4c0365531af20d9a08e6d7fbf020f528cfda2bbf42e"}
May 20 13:54:12.266738 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.266710    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71936781-21e0-4444-9bc4-c23d7840ae67","Type":"ContainerStarted","Data":"1d4c03ea3ecf082e739383d3e6b0a568ce5178d73ad5825e20eafbc725d31028"}
May 20 13:54:12.267104 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.266748    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71936781-21e0-4444-9bc4-c23d7840ae67","Type":"ContainerStarted","Data":"de69dc15d97a6cac84a078f59efb5bfb335b374d10d233ddcc8840d412f7fce5"}
May 20 13:54:12.267104 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.266760    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71936781-21e0-4444-9bc4-c23d7840ae67","Type":"ContainerStarted","Data":"69fd3f10fccd15b9f431305309d76708561ce80185663c92613728a117bd6c35"}
May 20 13:54:12.267104 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.266774    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71936781-21e0-4444-9bc4-c23d7840ae67","Type":"ContainerStarted","Data":"c395c52af7f986e69a950a33877a8557a300419b181a6bb05b6c39f7986972fc"}
May 20 13:54:12.267104 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.266787    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71936781-21e0-4444-9bc4-c23d7840ae67","Type":"ContainerStarted","Data":"09a44ea02c5f46e9aacca13f4f17b08943d0a59f3159f050d0a3203b291a6a07"}
May 20 13:54:12.267104 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.266801    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"71936781-21e0-4444-9bc4-c23d7840ae67","Type":"ContainerStarted","Data":"5b34829a245dbac0006d9fa51429148c1d661b84120673f375119563e31c09c1"}
May 20 13:54:12.295874 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.295803    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.295784175 podStartE2EDuration="2.295784175s" podCreationTimestamp="2026-05-20 13:54:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:54:12.293424668 +0000 UTC m=+232.430978885" watchObservedRunningTime="2026-05-20 13:54:12.295784175 +0000 UTC m=+232.433338396"
May 20 13:54:12.456959 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.456910    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:12.458053 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.458018    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66" gracePeriod=600
May 20 13:54:12.458181 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.458166    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy" containerID="cri-o://44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8" gracePeriod=600
May 20 13:54:12.458266 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.458243    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy-web" containerID="cri-o://321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3" gracePeriod=600
May 20 13:54:12.458353 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.458310    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="thanos-sidecar" containerID="cri-o://08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0" gracePeriod=600
May 20 13:54:12.458699 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.458383    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="config-reloader" containerID="cri-o://b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21" gracePeriod=600
May 20 13:54:12.458699 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:12.458445    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="prometheus" containerID="cri-o://f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9" gracePeriod=600
May 20 13:54:13.273296 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273263    2557 generic.go:358] "Generic (PLEG): container finished" podID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerID="6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66" exitCode=0
May 20 13:54:13.273296 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273291    2557 generic.go:358] "Generic (PLEG): container finished" podID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerID="44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8" exitCode=0
May 20 13:54:13.273296 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273297    2557 generic.go:358] "Generic (PLEG): container finished" podID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerID="08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0" exitCode=0
May 20 13:54:13.273698 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273303    2557 generic.go:358] "Generic (PLEG): container finished" podID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerID="b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21" exitCode=0
May 20 13:54:13.273698 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273309    2557 generic.go:358] "Generic (PLEG): container finished" podID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerID="f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9" exitCode=0
May 20 13:54:13.273698 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273342    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerDied","Data":"6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66"}
May 20 13:54:13.273698 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273390    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerDied","Data":"44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8"}
May 20 13:54:13.273698 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273407    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerDied","Data":"08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0"}
May 20 13:54:13.273698 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273422    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerDied","Data":"b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21"}
May 20 13:54:13.273698 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.273436    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerDied","Data":"f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9"}
May 20 13:54:13.702629 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.702610    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:13.858230 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858209    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858340 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858250    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-trusted-ca-bundle\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858340 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858269    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-kube-rbac-proxy\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858340 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858299    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-tls\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858340 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858329    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-db\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858520 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858364    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858520 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858403    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-rulefiles-0\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858520 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858427    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858520 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858454    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config-out\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858520 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858481    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-thanos-prometheus-http-client-file\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858755 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858534    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-grpc-tls\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858755 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858570    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-metrics-client-ca\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858755 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858597    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-metrics-client-certs\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858755 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858627    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmvrn\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-kube-api-access-gmvrn\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858755 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858665    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-tls-assets\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858755 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858690    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-web-config\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858755 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858714    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-serving-certs-ca-bundle\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.858755 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858742    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-kubelet-serving-ca-bundle\") pod \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\" (UID: \"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd\") "
May 20 13:54:13.859147 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.858943    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:13.859147 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.859064    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.859309 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.859283    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:13.859792 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.859517    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:54:13.860936 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.860837    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:13.860936 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.860876    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config" (OuterVolumeSpecName: "config") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:13.861082 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.861043    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:13.861082 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.861068    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:13.861191 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.861131    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:13.861653 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.861415    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:13.861653 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.861489    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:13.861653 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.861593    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:13.861879 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.861680    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-kube-api-access-gmvrn" (OuterVolumeSpecName: "kube-api-access-gmvrn") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "kube-api-access-gmvrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:13.862686 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.862654    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:13.863103 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.863071    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:13.863328 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.863310    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config-out" (OuterVolumeSpecName: "config-out") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:54:13.863518 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.863505    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:13.864006 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.863990    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:13.873173 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.873151    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-web-config" (OuterVolumeSpecName: "web-config") pod "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" (UID: "fcc0dc8f-3d23-4563-a8b6-5ca482a750bd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:13.960157 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960132    2557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-grpc-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960157 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960152    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-metrics-client-ca\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960162    2557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-metrics-client-certs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960171    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmvrn\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-kube-api-access-gmvrn\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960180    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-tls-assets\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960188    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-web-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960197    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960205    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960215    2557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960224    2557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-kube-rbac-proxy\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960232    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960241    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-db\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960249    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960259    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960268    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960274 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960277    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-config-out\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:13.960639 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:13.960285    2557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 13:54:14.280056 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.280028    2557 generic.go:358] "Generic (PLEG): container finished" podID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerID="321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3" exitCode=0
May 20 13:54:14.280349 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.280102    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerDied","Data":"321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3"}
May 20 13:54:14.280349 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.280131    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcc0dc8f-3d23-4563-a8b6-5ca482a750bd","Type":"ContainerDied","Data":"ea68b5461a08b3a4874e222bf817be4e546645886bbb3abcc9147b59bccd059e"}
May 20 13:54:14.280349 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.280147    2557 scope.go:117] "RemoveContainer" containerID="6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66"
May 20 13:54:14.280349 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.280157    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.288552 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.288533    2557 scope.go:117] "RemoveContainer" containerID="44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8"
May 20 13:54:14.295667 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.295650    2557 scope.go:117] "RemoveContainer" containerID="321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3"
May 20 13:54:14.305890 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.305829    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:14.305890 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.305877    2557 scope.go:117] "RemoveContainer" containerID="08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0"
May 20 13:54:14.311914 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.311846    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:14.313550 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.313533    2557 scope.go:117] "RemoveContainer" containerID="b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21"
May 20 13:54:14.319757 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.319741    2557 scope.go:117] "RemoveContainer" containerID="f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9"
May 20 13:54:14.326372 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.326356    2557 scope.go:117] "RemoveContainer" containerID="c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611"
May 20 13:54:14.332768 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.332753    2557 scope.go:117] "RemoveContainer" containerID="6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66"
May 20 13:54:14.333049 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:14.333030    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66\": container with ID starting with 6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66 not found: ID does not exist" containerID="6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66"
May 20 13:54:14.333108 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.333055    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66"} err="failed to get container status \"6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66\": rpc error: code = NotFound desc = could not find container \"6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66\": container with ID starting with 6fc745b53a9ddaec9e802371542eeb6ba7c1714bef01f4a9cd174984c06caa66 not found: ID does not exist"
May 20 13:54:14.333108 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.333072    2557 scope.go:117] "RemoveContainer" containerID="44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8"
May 20 13:54:14.333282 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:14.333268    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8\": container with ID starting with 44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8 not found: ID does not exist" containerID="44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8"
May 20 13:54:14.333315 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.333286    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8"} err="failed to get container status \"44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8\": rpc error: code = NotFound desc = could not find container \"44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8\": container with ID starting with 44cb7d4357ab58a98b6e53f1d3761e8a85697cdc8e79b84dc120895f1b6d37e8 not found: ID does not exist"
May 20 13:54:14.333315 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.333299    2557 scope.go:117] "RemoveContainer" containerID="321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3"
May 20 13:54:14.333541 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:14.333523    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3\": container with ID starting with 321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3 not found: ID does not exist" containerID="321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3"
May 20 13:54:14.333583 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.333546    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3"} err="failed to get container status \"321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3\": rpc error: code = NotFound desc = could not find container \"321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3\": container with ID starting with 321884d4c75057d2d6af7db9b451168e5f85356805c575e27b193f6ffff110c3 not found: ID does not exist"
May 20 13:54:14.333583 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.333562    2557 scope.go:117] "RemoveContainer" containerID="08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0"
May 20 13:54:14.333772 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:14.333755    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0\": container with ID starting with 08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0 not found: ID does not exist" containerID="08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0"
May 20 13:54:14.333831 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.333781    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0"} err="failed to get container status \"08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0\": rpc error: code = NotFound desc = could not find container \"08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0\": container with ID starting with 08ac286934fd1e927e4d43d372a00f1a2b41a35ab3d0198cf80e60ca87e489f0 not found: ID does not exist"
May 20 13:54:14.333831 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.333802    2557 scope.go:117] "RemoveContainer" containerID="b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21"
May 20 13:54:14.334083 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:14.334065    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21\": container with ID starting with b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21 not found: ID does not exist" containerID="b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21"
May 20 13:54:14.334121 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.334089    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21"} err="failed to get container status \"b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21\": rpc error: code = NotFound desc = could not find container \"b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21\": container with ID starting with b17dfa24d17cff5844954a29092ed5f01d5653187efe3ae8fa5b2f6385e3ae21 not found: ID does not exist"
May 20 13:54:14.334121 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.334105    2557 scope.go:117] "RemoveContainer" containerID="f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9"
May 20 13:54:14.334328 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:14.334314    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9\": container with ID starting with f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9 not found: ID does not exist" containerID="f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9"
May 20 13:54:14.334366 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.334330    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9"} err="failed to get container status \"f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9\": rpc error: code = NotFound desc = could not find container \"f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9\": container with ID starting with f1498caf385c44e4fb8595107742f753366d2e900fa643365ec2ff980cce0fe9 not found: ID does not exist"
May 20 13:54:14.334366 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.334343    2557 scope.go:117] "RemoveContainer" containerID="c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611"
May 20 13:54:14.334530 ip-10-0-139-248 kubenswrapper[2557]: E0520 13:54:14.334516    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611\": container with ID starting with c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611 not found: ID does not exist" containerID="c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611"
May 20 13:54:14.334583 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.334532    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611"} err="failed to get container status \"c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611\": rpc error: code = NotFound desc = could not find container \"c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611\": container with ID starting with c92b11e32974eead4bc989e1fda2d03de9a205d04891248a3da5e170fb3f2611 not found: ID does not exist"
May 20 13:54:14.336409 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336390    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:14.336703 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336690    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="init-config-reloader"
May 20 13:54:14.336739 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336704    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="init-config-reloader"
May 20 13:54:14.336739 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336718    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy-thanos"
May 20 13:54:14.336739 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336724    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy-thanos"
May 20 13:54:14.336739 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336735    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="prometheus"
May 20 13:54:14.336739 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336740    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="prometheus"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336752    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336757    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336765    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy-web"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336770    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy-web"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336777    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="config-reloader"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336782    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="config-reloader"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336789    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="thanos-sidecar"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336793    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="thanos-sidecar"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336845    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="thanos-sidecar"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336871    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336880    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy-thanos"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336888    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="config-reloader"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336895    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="kube-rbac-proxy-web"
May 20 13:54:14.336906 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.336901    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" containerName="prometheus"
May 20 13:54:14.342035 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.342020    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.344935 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.344917    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 20 13:54:14.345378 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.345361    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 20 13:54:14.345578 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.345565    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 20 13:54:14.346021 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.346004    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 20 13:54:14.346890 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.346875    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 20 13:54:14.347098 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.347079    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-frs7k\""
May 20 13:54:14.347212 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.347121    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 20 13:54:14.347212 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.347153    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 20 13:54:14.347212 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.347187    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 20 13:54:14.347212 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.347205    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5419umi8scg9s\""
May 20 13:54:14.347405 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.347158    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 20 13:54:14.347676 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.347569    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 20 13:54:14.348497 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.348476    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 20 13:54:14.349160 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.349140    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 20 13:54:14.350618 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.350598    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 20 13:54:14.354572 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.354551    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:14.404683 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.404664    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc0dc8f-3d23-4563-a8b6-5ca482a750bd" path="/var/lib/kubelet/pods/fcc0dc8f-3d23-4563-a8b6-5ca482a750bd/volumes"
May 20 13:54:14.464164 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464234 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70d7745b-dc43-4d35-8506-00a0df10b98b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464234 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464190    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464234 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464211    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464328 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464236    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-web-config\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464328 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464258    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464385 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464331    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464385 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464357    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70d7745b-dc43-4d35-8506-00a0df10b98b-config-out\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464442 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464391    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464442 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464499 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464446    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464499 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464487    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464557 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464557 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464535    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464612 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464556    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70d7745b-dc43-4d35-8506-00a0df10b98b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464612 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-config\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464612 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69wb2\" (UniqueName: \"kubernetes.io/projected/70d7745b-dc43-4d35-8506-00a0df10b98b-kube-api-access-69wb2\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.464612 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.464609    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.565896 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.565830    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70d7745b-dc43-4d35-8506-00a0df10b98b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.565896 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.565887    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566051 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.565908    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566051 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.565932    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-web-config\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566051 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.565951    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566051 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.565971    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566051 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566001    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70d7745b-dc43-4d35-8506-00a0df10b98b-config-out\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566051 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566035    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566064    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566103    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566150    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566181    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566213    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566242    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70d7745b-dc43-4d35-8506-00a0df10b98b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566254    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70d7745b-dc43-4d35-8506-00a0df10b98b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566306    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-config\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566334 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566335    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-69wb2\" (UniqueName: \"kubernetes.io/projected/70d7745b-dc43-4d35-8506-00a0df10b98b-kube-api-access-69wb2\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566770 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566364    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566770 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566411    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.566883 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.566802    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.567179 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.567146    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.569262 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.569230    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-web-config\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.569368 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.569328    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.569368 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.569354    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.569470 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.569365    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.570520 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.569844    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70d7745b-dc43-4d35-8506-00a0df10b98b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.570520 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.570163    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.570520 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.570307    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.570736 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.570708    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.570893 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.570838    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70d7745b-dc43-4d35-8506-00a0df10b98b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.570988 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.570943    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70d7745b-dc43-4d35-8506-00a0df10b98b-config-out\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.571929 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.571896    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.572124 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.572105    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.572211 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.572187    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-config\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.572455 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.572437    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70d7745b-dc43-4d35-8506-00a0df10b98b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.577868 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.577829    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69wb2\" (UniqueName: \"kubernetes.io/projected/70d7745b-dc43-4d35-8506-00a0df10b98b-kube-api-access-69wb2\") pod \"prometheus-k8s-0\" (UID: \"70d7745b-dc43-4d35-8506-00a0df10b98b\") " pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.651145 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.651121    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:54:14.790987 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:14.790964    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 20 13:54:14.792703 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:54:14.792679    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d7745b_dc43_4d35_8506_00a0df10b98b.slice/crio-e912fd02d971938942ed008c53f88a4547b00fa918f471ad72c70c5dafab41bc WatchSource:0}: Error finding container e912fd02d971938942ed008c53f88a4547b00fa918f471ad72c70c5dafab41bc: Status 404 returned error can't find the container with id e912fd02d971938942ed008c53f88a4547b00fa918f471ad72c70c5dafab41bc
May 20 13:54:15.286001 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:15.285915    2557 generic.go:358] "Generic (PLEG): container finished" podID="70d7745b-dc43-4d35-8506-00a0df10b98b" containerID="e60ea60d8ed6a03fbf1b872dc6dcb132755cf1bbc90f533123c78e640c18deb0" exitCode=0
May 20 13:54:15.286001 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:15.285971    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70d7745b-dc43-4d35-8506-00a0df10b98b","Type":"ContainerDied","Data":"e60ea60d8ed6a03fbf1b872dc6dcb132755cf1bbc90f533123c78e640c18deb0"}
May 20 13:54:15.286001 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:15.285996    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70d7745b-dc43-4d35-8506-00a0df10b98b","Type":"ContainerStarted","Data":"e912fd02d971938942ed008c53f88a4547b00fa918f471ad72c70c5dafab41bc"}
May 20 13:54:16.293234 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:16.293204    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70d7745b-dc43-4d35-8506-00a0df10b98b","Type":"ContainerStarted","Data":"83202d3bad015bbb0b7a5b34caf45cf8488cd3e51496ab2813e9c96fe3d632bd"}
May 20 13:54:16.293570 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:16.293241    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70d7745b-dc43-4d35-8506-00a0df10b98b","Type":"ContainerStarted","Data":"14c9eb5a695ee1cdfe0d48a4fce1df648edee59592db0193da3172aa155aefb5"}
May 20 13:54:16.293570 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:16.293252    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70d7745b-dc43-4d35-8506-00a0df10b98b","Type":"ContainerStarted","Data":"2e73745f8fdee2728d85dfa7e743dd25064c5e2d98b0af86ede5ce440e172ff7"}
May 20 13:54:16.293570 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:16.293260    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70d7745b-dc43-4d35-8506-00a0df10b98b","Type":"ContainerStarted","Data":"26ff56b6acaeec1104616b0ec1804708675046f9898bd6b9e25f4a6811d06c0e"}
May 20 13:54:16.293570 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:16.293269    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70d7745b-dc43-4d35-8506-00a0df10b98b","Type":"ContainerStarted","Data":"486298d035ef72effe7e81a892ba4b8e28593a0b439429d241b17322967a9089"}
May 20 13:54:16.293570 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:16.293277    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70d7745b-dc43-4d35-8506-00a0df10b98b","Type":"ContainerStarted","Data":"a0fc07b3a66a1a50dfee9a7dc2c727506459edc1fc7d1f3ef59a83dfe84a69c7"}
May 20 13:54:16.323758 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:16.323712    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.323698382 podStartE2EDuration="2.323698382s" podCreationTimestamp="2026-05-20 13:54:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:54:16.321273596 +0000 UTC m=+236.458827859" watchObservedRunningTime="2026-05-20 13:54:16.323698382 +0000 UTC m=+236.461252601"
May 20 13:54:19.651878 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:54:19.651824    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:55:14.652026 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:14.651986    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:55:14.668870 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:14.668810    2557 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:55:15.488019 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:15.487997    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 20 13:55:20.289955 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:20.289928    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 13:55:20.289955 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:20.289954    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 13:55:20.294463 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:20.294444    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 13:55:20.294582 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:20.294549    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 13:55:20.298436 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:20.298421    2557 kubelet.go:1628] "Image garbage collection succeeded"
May 20 13:55:57.925707 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:57.925664    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-tcj8j"]
May 20 13:55:57.929256 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:57.929232    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:57.932075 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:57.932055    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 13:55:57.937781 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:57.937758    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tcj8j"]
May 20 13:55:58.017401 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.017373    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e804929-2290-4e4c-b3d6-c9ea04e60e40-original-pull-secret\") pod \"global-pull-secret-syncer-tcj8j\" (UID: \"0e804929-2290-4e4c-b3d6-c9ea04e60e40\") " pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.017516 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.017445    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0e804929-2290-4e4c-b3d6-c9ea04e60e40-kubelet-config\") pod \"global-pull-secret-syncer-tcj8j\" (UID: \"0e804929-2290-4e4c-b3d6-c9ea04e60e40\") " pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.017516 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.017491    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0e804929-2290-4e4c-b3d6-c9ea04e60e40-dbus\") pod \"global-pull-secret-syncer-tcj8j\" (UID: \"0e804929-2290-4e4c-b3d6-c9ea04e60e40\") " pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.118664 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.118637    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e804929-2290-4e4c-b3d6-c9ea04e60e40-original-pull-secret\") pod \"global-pull-secret-syncer-tcj8j\" (UID: \"0e804929-2290-4e4c-b3d6-c9ea04e60e40\") " pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.118775 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.118701    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0e804929-2290-4e4c-b3d6-c9ea04e60e40-kubelet-config\") pod \"global-pull-secret-syncer-tcj8j\" (UID: \"0e804929-2290-4e4c-b3d6-c9ea04e60e40\") " pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.118775 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.118731    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0e804929-2290-4e4c-b3d6-c9ea04e60e40-dbus\") pod \"global-pull-secret-syncer-tcj8j\" (UID: \"0e804929-2290-4e4c-b3d6-c9ea04e60e40\") " pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.118986 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.118842    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0e804929-2290-4e4c-b3d6-c9ea04e60e40-kubelet-config\") pod \"global-pull-secret-syncer-tcj8j\" (UID: \"0e804929-2290-4e4c-b3d6-c9ea04e60e40\") " pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.118986 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.118911    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0e804929-2290-4e4c-b3d6-c9ea04e60e40-dbus\") pod \"global-pull-secret-syncer-tcj8j\" (UID: \"0e804929-2290-4e4c-b3d6-c9ea04e60e40\") " pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.121115 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.121094    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0e804929-2290-4e4c-b3d6-c9ea04e60e40-original-pull-secret\") pod \"global-pull-secret-syncer-tcj8j\" (UID: \"0e804929-2290-4e4c-b3d6-c9ea04e60e40\") " pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.239391 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.239335    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tcj8j"
May 20 13:55:58.357042 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.357018    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tcj8j"]
May 20 13:55:58.359338 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:55:58.359309    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e804929_2290_4e4c_b3d6_c9ea04e60e40.slice/crio-0747a39f45a66b1a00b82b1253df0529b0e1269f3c5d283867ffb53c1c6bd9fd WatchSource:0}: Error finding container 0747a39f45a66b1a00b82b1253df0529b0e1269f3c5d283867ffb53c1c6bd9fd: Status 404 returned error can't find the container with id 0747a39f45a66b1a00b82b1253df0529b0e1269f3c5d283867ffb53c1c6bd9fd
May 20 13:55:58.361063 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.361047    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:55:58.607820 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:55:58.607794    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tcj8j" event={"ID":"0e804929-2290-4e4c-b3d6-c9ea04e60e40","Type":"ContainerStarted","Data":"0747a39f45a66b1a00b82b1253df0529b0e1269f3c5d283867ffb53c1c6bd9fd"}
May 20 13:56:02.622243 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:56:02.622201    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tcj8j" event={"ID":"0e804929-2290-4e4c-b3d6-c9ea04e60e40","Type":"ContainerStarted","Data":"e779d5d029e4eec8b705456a41f1462e85189f6c2b9b2c90caf1085175c0e1dd"}
May 20 13:56:02.638813 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:56:02.638758    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tcj8j" podStartSLOduration=2.143618121 podStartE2EDuration="5.638743447s" podCreationTimestamp="2026-05-20 13:55:57 +0000 UTC" firstStartedPulling="2026-05-20 13:55:58.361170232 +0000 UTC m=+338.498724428" lastFinishedPulling="2026-05-20 13:56:01.856295557 +0000 UTC m=+341.993849754" observedRunningTime="2026-05-20 13:56:02.637833736 +0000 UTC m=+342.775387954" watchObservedRunningTime="2026-05-20 13:56:02.638743447 +0000 UTC m=+342.776297664"
May 20 13:59:19.219820 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.219750    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-x6tm9"]
May 20 13:59:19.222291 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.222274    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6tm9"
May 20 13:59:19.225046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.225021    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 13:59:19.225046 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.225039    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 13:59:19.225201 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.225063    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 20 13:59:19.226164 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.226145    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-nwmm8\""
May 20 13:59:19.230726 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.230703    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x6tm9"]
May 20 13:59:19.273602 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.273578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c29e6f-9729-4b30-8ccd-3741d12f722d-model-cache\") pod \"s3-init-x6tm9\" (UID: \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\") " pod="kserve/s3-init-x6tm9"
May 20 13:59:19.273702 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.273634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2b94\" (UniqueName: \"kubernetes.io/projected/a7c29e6f-9729-4b30-8ccd-3741d12f722d-kube-api-access-b2b94\") pod \"s3-init-x6tm9\" (UID: \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\") " pod="kserve/s3-init-x6tm9"
May 20 13:59:19.374684 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.374660    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c29e6f-9729-4b30-8ccd-3741d12f722d-model-cache\") pod \"s3-init-x6tm9\" (UID: \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\") " pod="kserve/s3-init-x6tm9"
May 20 13:59:19.374791 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.374722    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b2b94\" (UniqueName: \"kubernetes.io/projected/a7c29e6f-9729-4b30-8ccd-3741d12f722d-kube-api-access-b2b94\") pod \"s3-init-x6tm9\" (UID: \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\") " pod="kserve/s3-init-x6tm9"
May 20 13:59:19.374988 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.374971    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c29e6f-9729-4b30-8ccd-3741d12f722d-model-cache\") pod \"s3-init-x6tm9\" (UID: \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\") " pod="kserve/s3-init-x6tm9"
May 20 13:59:19.384389 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.384367    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2b94\" (UniqueName: \"kubernetes.io/projected/a7c29e6f-9729-4b30-8ccd-3741d12f722d-kube-api-access-b2b94\") pod \"s3-init-x6tm9\" (UID: \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\") " pod="kserve/s3-init-x6tm9"
May 20 13:59:19.547702 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.547640    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6tm9"
May 20 13:59:19.673665 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:19.673640    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-x6tm9"]
May 20 13:59:19.676518 ip-10-0-139-248 kubenswrapper[2557]: W0520 13:59:19.676487    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7c29e6f_9729_4b30_8ccd_3741d12f722d.slice/crio-2ee11b047ffa19ff66896b84b5f748e2a136306e843d206c1e061d8cb697cbdb WatchSource:0}: Error finding container 2ee11b047ffa19ff66896b84b5f748e2a136306e843d206c1e061d8cb697cbdb: Status 404 returned error can't find the container with id 2ee11b047ffa19ff66896b84b5f748e2a136306e843d206c1e061d8cb697cbdb
May 20 13:59:20.253543 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:20.253507    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6tm9" event={"ID":"a7c29e6f-9729-4b30-8ccd-3741d12f722d","Type":"ContainerStarted","Data":"2ee11b047ffa19ff66896b84b5f748e2a136306e843d206c1e061d8cb697cbdb"}
May 20 13:59:24.272804 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:24.272757    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6tm9" event={"ID":"a7c29e6f-9729-4b30-8ccd-3741d12f722d","Type":"ContainerStarted","Data":"013c46e2fbf2284eddd0fdaa96490df62a3d4cc4c0873daf5d34ff6b24c8fe29"}
May 20 13:59:25.277467 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:25.277425    2557 generic.go:358] "Generic (PLEG): container finished" podID="a7c29e6f-9729-4b30-8ccd-3741d12f722d" containerID="013c46e2fbf2284eddd0fdaa96490df62a3d4cc4c0873daf5d34ff6b24c8fe29" exitCode=0
May 20 13:59:25.277923 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:25.277472    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6tm9" event={"ID":"a7c29e6f-9729-4b30-8ccd-3741d12f722d","Type":"ContainerDied","Data":"013c46e2fbf2284eddd0fdaa96490df62a3d4cc4c0873daf5d34ff6b24c8fe29"}
May 20 13:59:29.625767 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:29.625744    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 20 13:59:30.295841 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:30.295805    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6tm9" event={"ID":"a7c29e6f-9729-4b30-8ccd-3741d12f722d","Type":"ContainerStarted","Data":"382ba845bff7828718f7ee5af78a33c91108c85a3ea9ac289ea8d5c195909af6"}
May 20 13:59:30.313168 ip-10-0-139-248 kubenswrapper[2557]: I0520 13:59:30.313121    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-x6tm9" podStartSLOduration=1.368762958 podStartE2EDuration="11.313105004s" podCreationTimestamp="2026-05-20 13:59:19 +0000 UTC" firstStartedPulling="2026-05-20 13:59:19.678197412 +0000 UTC m=+539.815751614" lastFinishedPulling="2026-05-20 13:59:29.622539462 +0000 UTC m=+549.760093660" observedRunningTime="2026-05-20 13:59:30.312446098 +0000 UTC m=+550.450000316" watchObservedRunningTime="2026-05-20 13:59:30.313105004 +0000 UTC m=+550.450659224"
May 20 14:00:20.319371 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:20.319344    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:00:20.321128 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:20.321110    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:00:20.323443 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:20.323426    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:00:20.325210 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:20.325190    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:00:34.488127 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:34.488099    2557 generic.go:358] "Generic (PLEG): container finished" podID="a7c29e6f-9729-4b30-8ccd-3741d12f722d" containerID="382ba845bff7828718f7ee5af78a33c91108c85a3ea9ac289ea8d5c195909af6" exitCode=0
May 20 14:00:34.488414 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:34.488172    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6tm9" event={"ID":"a7c29e6f-9729-4b30-8ccd-3741d12f722d","Type":"ContainerDied","Data":"382ba845bff7828718f7ee5af78a33c91108c85a3ea9ac289ea8d5c195909af6"}
May 20 14:00:35.621987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:35.621965    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6tm9"
May 20 14:00:35.647878 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:35.647836    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c29e6f-9729-4b30-8ccd-3741d12f722d-model-cache\") pod \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\" (UID: \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\") "
May 20 14:00:35.667172 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:35.667147    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c29e6f-9729-4b30-8ccd-3741d12f722d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7c29e6f-9729-4b30-8ccd-3741d12f722d" (UID: "a7c29e6f-9729-4b30-8ccd-3741d12f722d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:00:35.748719 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:35.748672    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2b94\" (UniqueName: \"kubernetes.io/projected/a7c29e6f-9729-4b30-8ccd-3741d12f722d-kube-api-access-b2b94\") pod \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\" (UID: \"a7c29e6f-9729-4b30-8ccd-3741d12f722d\") "
May 20 14:00:35.748830 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:35.748792    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c29e6f-9729-4b30-8ccd-3741d12f722d-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:00:35.750636 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:35.750613    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c29e6f-9729-4b30-8ccd-3741d12f722d-kube-api-access-b2b94" (OuterVolumeSpecName: "kube-api-access-b2b94") pod "a7c29e6f-9729-4b30-8ccd-3741d12f722d" (UID: "a7c29e6f-9729-4b30-8ccd-3741d12f722d"). InnerVolumeSpecName "kube-api-access-b2b94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:00:35.849125 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:35.849102    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2b94\" (UniqueName: \"kubernetes.io/projected/a7c29e6f-9729-4b30-8ccd-3741d12f722d-kube-api-access-b2b94\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:00:36.495210 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:36.495185    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-x6tm9" event={"ID":"a7c29e6f-9729-4b30-8ccd-3741d12f722d","Type":"ContainerDied","Data":"2ee11b047ffa19ff66896b84b5f748e2a136306e843d206c1e061d8cb697cbdb"}
May 20 14:00:36.495210 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:36.495206    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-x6tm9"
May 20 14:00:36.495356 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:00:36.495210    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ee11b047ffa19ff66896b84b5f748e2a136306e843d206c1e061d8cb697cbdb"
May 20 14:01:14.201340 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.201298    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-5mc94"]
May 20 14:01:14.202763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.201756    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c29e6f-9729-4b30-8ccd-3741d12f722d" containerName="s3-init"
May 20 14:01:14.202763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.201772    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c29e6f-9729-4b30-8ccd-3741d12f722d" containerName="s3-init"
May 20 14:01:14.202763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.201791    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c29e6f-9729-4b30-8ccd-3741d12f722d" containerName="download-hf-model"
May 20 14:01:14.202763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.201800    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c29e6f-9729-4b30-8ccd-3741d12f722d" containerName="download-hf-model"
May 20 14:01:14.202763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.201873    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="a7c29e6f-9729-4b30-8ccd-3741d12f722d" containerName="s3-init"
May 20 14:01:14.204048 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.204029    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:14.207149 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.207120    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 14:01:14.207284 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.207195    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
May 20 14:01:14.207284 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.207195    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-nwmm8\""
May 20 14:01:14.209348 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.209324    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 14:01:14.214640 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.214615    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-5mc94"]
May 20 14:01:14.314687 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.314660    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-model-cache\") pod \"s3-tls-init-custom-5mc94\" (UID: \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\") " pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:14.314788 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.314712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kr2l\" (UniqueName: \"kubernetes.io/projected/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-kube-api-access-7kr2l\") pod \"s3-tls-init-custom-5mc94\" (UID: \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\") " pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:14.415564 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.415529    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7kr2l\" (UniqueName: \"kubernetes.io/projected/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-kube-api-access-7kr2l\") pod \"s3-tls-init-custom-5mc94\" (UID: \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\") " pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:14.415683 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.415615    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-model-cache\") pod \"s3-tls-init-custom-5mc94\" (UID: \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\") " pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:14.416106 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.416079    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-model-cache\") pod \"s3-tls-init-custom-5mc94\" (UID: \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\") " pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:14.424923 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.424899    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kr2l\" (UniqueName: \"kubernetes.io/projected/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-kube-api-access-7kr2l\") pod \"s3-tls-init-custom-5mc94\" (UID: \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\") " pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:14.523044 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.522989    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:14.655902 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.655849    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-5mc94"]
May 20 14:01:14.658651 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:01:14.658624    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a6aa934_7bc4_4a4c_884e_c0118ed51c5c.slice/crio-4338029ec5a7a99655bfe13639ad38ece4a1f2f3f4d04a06457e3ef574b224c4 WatchSource:0}: Error finding container 4338029ec5a7a99655bfe13639ad38ece4a1f2f3f4d04a06457e3ef574b224c4: Status 404 returned error can't find the container with id 4338029ec5a7a99655bfe13639ad38ece4a1f2f3f4d04a06457e3ef574b224c4
May 20 14:01:14.660445 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:14.660431    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:01:15.622154 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:15.619938    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5mc94" event={"ID":"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c","Type":"ContainerStarted","Data":"07c693d084f551e9620300002bd91429b49fef157f8b196d9128db4eaf415d01"}
May 20 14:01:15.622154 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:15.619987    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5mc94" event={"ID":"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c","Type":"ContainerStarted","Data":"4338029ec5a7a99655bfe13639ad38ece4a1f2f3f4d04a06457e3ef574b224c4"}
May 20 14:01:16.624195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:16.624129    2557 generic.go:358] "Generic (PLEG): container finished" podID="9a6aa934-7bc4-4a4c-884e-c0118ed51c5c" containerID="07c693d084f551e9620300002bd91429b49fef157f8b196d9128db4eaf415d01" exitCode=0
May 20 14:01:16.624490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:16.624214    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5mc94" event={"ID":"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c","Type":"ContainerDied","Data":"07c693d084f551e9620300002bd91429b49fef157f8b196d9128db4eaf415d01"}
May 20 14:01:17.628938 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:17.628895    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5mc94" event={"ID":"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c","Type":"ContainerStarted","Data":"83e10090f27c7be807c148b92160c331ad4827df6d33c4fea050090de9b9c382"}
May 20 14:01:17.656978 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:17.656923    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-5mc94" podStartSLOduration=3.65690571 podStartE2EDuration="3.65690571s" podCreationTimestamp="2026-05-20 14:01:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:01:17.653973766 +0000 UTC m=+657.791528010" watchObservedRunningTime="2026-05-20 14:01:17.65690571 +0000 UTC m=+657.794459931"
May 20 14:01:23.649588 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:23.649556    2557 generic.go:358] "Generic (PLEG): container finished" podID="9a6aa934-7bc4-4a4c-884e-c0118ed51c5c" containerID="83e10090f27c7be807c148b92160c331ad4827df6d33c4fea050090de9b9c382" exitCode=0
May 20 14:01:23.649958 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:23.649637    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5mc94" event={"ID":"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c","Type":"ContainerDied","Data":"83e10090f27c7be807c148b92160c331ad4827df6d33c4fea050090de9b9c382"}
May 20 14:01:24.826965 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:24.826943    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:24.898109 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:24.898085    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-model-cache\") pod \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\" (UID: \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\") "
May 20 14:01:24.898225 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:24.898125    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kr2l\" (UniqueName: \"kubernetes.io/projected/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-kube-api-access-7kr2l\") pod \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\" (UID: \"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c\") "
May 20 14:01:24.900259 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:24.900224    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-kube-api-access-7kr2l" (OuterVolumeSpecName: "kube-api-access-7kr2l") pod "9a6aa934-7bc4-4a4c-884e-c0118ed51c5c" (UID: "9a6aa934-7bc4-4a4c-884e-c0118ed51c5c"). InnerVolumeSpecName "kube-api-access-7kr2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:01:24.919964 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:24.919938    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a6aa934-7bc4-4a4c-884e-c0118ed51c5c" (UID: "9a6aa934-7bc4-4a4c-884e-c0118ed51c5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:01:24.999606 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:24.999582    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:01:24.999606 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:24.999604    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kr2l\" (UniqueName: \"kubernetes.io/projected/9a6aa934-7bc4-4a4c-884e-c0118ed51c5c-kube-api-access-7kr2l\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:01:25.657546 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:25.657515    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-5mc94"
May 20 14:01:25.657546 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:25.657537    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-5mc94" event={"ID":"9a6aa934-7bc4-4a4c-884e-c0118ed51c5c","Type":"ContainerDied","Data":"4338029ec5a7a99655bfe13639ad38ece4a1f2f3f4d04a06457e3ef574b224c4"}
May 20 14:01:25.657780 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:25.657566    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4338029ec5a7a99655bfe13639ad38ece4a1f2f3f4d04a06457e3ef574b224c4"
May 20 14:01:28.278647 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.278604    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-crtgt"]
May 20 14:01:28.279153 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.278994    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6aa934-7bc4-4a4c-884e-c0118ed51c5c" containerName="s3-tls-init-custom"
May 20 14:01:28.279153 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.279008    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6aa934-7bc4-4a4c-884e-c0118ed51c5c" containerName="s3-tls-init-custom"
May 20 14:01:28.279153 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.279020    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6aa934-7bc4-4a4c-884e-c0118ed51c5c" containerName="download-hf-model"
May 20 14:01:28.279153 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.279028    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6aa934-7bc4-4a4c-884e-c0118ed51c5c" containerName="download-hf-model"
May 20 14:01:28.279153 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.279109    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="9a6aa934-7bc4-4a4c-884e-c0118ed51c5c" containerName="s3-tls-init-custom"
May 20 14:01:28.281561 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.281545    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:01:28.284380 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.284361    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 14:01:28.284508 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.284390    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-nwmm8\""
May 20 14:01:28.284508 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.284361    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 14:01:28.285455 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.285435    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
May 20 14:01:28.290580 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.290560    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-crtgt"]
May 20 14:01:28.326743 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.326719    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5wj\" (UniqueName: \"kubernetes.io/projected/e9c7c931-2114-4774-8579-f89f4efb905f-kube-api-access-xg5wj\") pod \"s3-tls-init-serving-crtgt\" (UID: \"e9c7c931-2114-4774-8579-f89f4efb905f\") " pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:01:28.326878 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.326757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c7c931-2114-4774-8579-f89f4efb905f-model-cache\") pod \"s3-tls-init-serving-crtgt\" (UID: \"e9c7c931-2114-4774-8579-f89f4efb905f\") " pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:01:28.427507 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.427480    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5wj\" (UniqueName: \"kubernetes.io/projected/e9c7c931-2114-4774-8579-f89f4efb905f-kube-api-access-xg5wj\") pod \"s3-tls-init-serving-crtgt\" (UID: \"e9c7c931-2114-4774-8579-f89f4efb905f\") " pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:01:28.427612 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.427521    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c7c931-2114-4774-8579-f89f4efb905f-model-cache\") pod \"s3-tls-init-serving-crtgt\" (UID: \"e9c7c931-2114-4774-8579-f89f4efb905f\") " pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:01:28.427954 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.427937    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c7c931-2114-4774-8579-f89f4efb905f-model-cache\") pod \"s3-tls-init-serving-crtgt\" (UID: \"e9c7c931-2114-4774-8579-f89f4efb905f\") " pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:01:28.437689 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.437667    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5wj\" (UniqueName: \"kubernetes.io/projected/e9c7c931-2114-4774-8579-f89f4efb905f-kube-api-access-xg5wj\") pod \"s3-tls-init-serving-crtgt\" (UID: \"e9c7c931-2114-4774-8579-f89f4efb905f\") " pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:01:28.604599 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.604572    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:01:28.729062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:28.729032    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-crtgt"]
May 20 14:01:28.732092 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:01:28.732067    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9c7c931_2114_4774_8579_f89f4efb905f.slice/crio-167be7f0e370d4d502a28fc401c364b19edc8c93d6a2e0e24c0d38c03091d3dc WatchSource:0}: Error finding container 167be7f0e370d4d502a28fc401c364b19edc8c93d6a2e0e24c0d38c03091d3dc: Status 404 returned error can't find the container with id 167be7f0e370d4d502a28fc401c364b19edc8c93d6a2e0e24c0d38c03091d3dc
May 20 14:01:29.673876 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:29.673824    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-crtgt" event={"ID":"e9c7c931-2114-4774-8579-f89f4efb905f","Type":"ContainerStarted","Data":"3075e007f4703c0cdc2de6f74831967028e5259212bbf3cdd6f58a470ab5c918"}
May 20 14:01:29.674365 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:01:29.674341    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-crtgt" event={"ID":"e9c7c931-2114-4774-8579-f89f4efb905f","Type":"ContainerStarted","Data":"167be7f0e370d4d502a28fc401c364b19edc8c93d6a2e0e24c0d38c03091d3dc"}
May 20 14:02:38.889001 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:38.888969    2557 generic.go:358] "Generic (PLEG): container finished" podID="e9c7c931-2114-4774-8579-f89f4efb905f" containerID="3075e007f4703c0cdc2de6f74831967028e5259212bbf3cdd6f58a470ab5c918" exitCode=0
May 20 14:02:38.889386 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:38.889040    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-crtgt" event={"ID":"e9c7c931-2114-4774-8579-f89f4efb905f","Type":"ContainerDied","Data":"3075e007f4703c0cdc2de6f74831967028e5259212bbf3cdd6f58a470ab5c918"}
May 20 14:02:38.898062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:38.898045    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
May 20 14:02:39.895826 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:39.895793    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-crtgt" event={"ID":"e9c7c931-2114-4774-8579-f89f4efb905f","Type":"ContainerStarted","Data":"d4566cb1675b767f9920b6853ce8a2f72ef5ebee78a6cb5e6f0dac04594558fa"}
May 20 14:02:39.915226 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:39.915183    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-crtgt" podStartSLOduration=71.915169198 podStartE2EDuration="1m11.915169198s" podCreationTimestamp="2026-05-20 14:01:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:02:39.913900156 +0000 UTC m=+740.051454375" watchObservedRunningTime="2026-05-20 14:02:39.915169198 +0000 UTC m=+740.052723416"
May 20 14:02:46.919456 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:46.919420    2557 generic.go:358] "Generic (PLEG): container finished" podID="e9c7c931-2114-4774-8579-f89f4efb905f" containerID="d4566cb1675b767f9920b6853ce8a2f72ef5ebee78a6cb5e6f0dac04594558fa" exitCode=0
May 20 14:02:46.919822 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:46.919477    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-crtgt" event={"ID":"e9c7c931-2114-4774-8579-f89f4efb905f","Type":"ContainerDied","Data":"d4566cb1675b767f9920b6853ce8a2f72ef5ebee78a6cb5e6f0dac04594558fa"}
May 20 14:02:48.060370 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.060349    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:02:48.136220 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.136185    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c7c931-2114-4774-8579-f89f4efb905f-model-cache\") pod \"e9c7c931-2114-4774-8579-f89f4efb905f\" (UID: \"e9c7c931-2114-4774-8579-f89f4efb905f\") "
May 20 14:02:48.136381 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.136228    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg5wj\" (UniqueName: \"kubernetes.io/projected/e9c7c931-2114-4774-8579-f89f4efb905f-kube-api-access-xg5wj\") pod \"e9c7c931-2114-4774-8579-f89f4efb905f\" (UID: \"e9c7c931-2114-4774-8579-f89f4efb905f\") "
May 20 14:02:48.138429 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.138403    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c7c931-2114-4774-8579-f89f4efb905f-kube-api-access-xg5wj" (OuterVolumeSpecName: "kube-api-access-xg5wj") pod "e9c7c931-2114-4774-8579-f89f4efb905f" (UID: "e9c7c931-2114-4774-8579-f89f4efb905f"). InnerVolumeSpecName "kube-api-access-xg5wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:02:48.155700 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.155656    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9c7c931-2114-4774-8579-f89f4efb905f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9c7c931-2114-4774-8579-f89f4efb905f" (UID: "e9c7c931-2114-4774-8579-f89f4efb905f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:02:48.237714 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.237628    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9c7c931-2114-4774-8579-f89f4efb905f-model-cache\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:02:48.237714 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.237661    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5wj\" (UniqueName: \"kubernetes.io/projected/e9c7c931-2114-4774-8579-f89f4efb905f-kube-api-access-xg5wj\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:02:48.926716 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.926686    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-crtgt"
May 20 14:02:48.926902 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.926714    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-crtgt" event={"ID":"e9c7c931-2114-4774-8579-f89f4efb905f","Type":"ContainerDied","Data":"167be7f0e370d4d502a28fc401c364b19edc8c93d6a2e0e24c0d38c03091d3dc"}
May 20 14:02:48.926902 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:02:48.926742    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="167be7f0e370d4d502a28fc401c364b19edc8c93d6a2e0e24c0d38c03091d3dc"
May 20 14:03:00.358071 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.358031    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"]
May 20 14:03:00.358464 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.358396    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c7c931-2114-4774-8579-f89f4efb905f" containerName="download-hf-model"
May 20 14:03:00.358464 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.358408    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c7c931-2114-4774-8579-f89f4efb905f" containerName="download-hf-model"
May 20 14:03:00.358464 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.358420    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9c7c931-2114-4774-8579-f89f4efb905f" containerName="s3-tls-init-serving"
May 20 14:03:00.358464 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.358426    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9c7c931-2114-4774-8579-f89f4efb905f" containerName="s3-tls-init-serving"
May 20 14:03:00.358664 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.358496    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9c7c931-2114-4774-8579-f89f4efb905f" containerName="s3-tls-init-serving"
May 20 14:03:00.361706 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.361691    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.365192 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.365166    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:03:00.365332 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.365267    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
May 20 14:03:00.365416 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.365347    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:03:00.365416 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.365379    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2mlds\""
May 20 14:03:00.366558 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.366536    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 14:03:00.366676 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.366539    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
May 20 14:03:00.366676 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.366568    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:03:00.374913 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.374891    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"]
May 20 14:03:00.440747 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.440723    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.440933 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.440762    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzv9k\" (UniqueName: \"kubernetes.io/projected/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kube-api-access-kzv9k\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.440933 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.440799    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.440933 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.440849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.441079 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.440928    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.541365 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.541312    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.541536 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.541383    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.541536 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.541407    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.541662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.541625    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.541721 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.541696    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kzv9k\" (UniqueName: \"kubernetes.io/projected/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kube-api-access-kzv9k\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.541816 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.541791    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.542272 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.542252    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.542388 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.542280    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.543769 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.543749    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.550301 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.550274    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzv9k\" (UniqueName: \"kubernetes.io/projected/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kube-api-access-kzv9k\") pod \"isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.671990 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.671891    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:00.804580 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.804556    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"]
May 20 14:03:00.807531 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:03:00.807503    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5c4776_7e80_4fe4_9700_1e6ba0f41c7f.slice/crio-a7d05f071b68b83d057ed2a4a9b90d59241d5da4787ef8c2c7ea8bea1d7bb1cb WatchSource:0}: Error finding container a7d05f071b68b83d057ed2a4a9b90d59241d5da4787ef8c2c7ea8bea1d7bb1cb: Status 404 returned error can't find the container with id a7d05f071b68b83d057ed2a4a9b90d59241d5da4787ef8c2c7ea8bea1d7bb1cb
May 20 14:03:00.965730 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.965644    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerStarted","Data":"e7ed56eaad95472f15f56e574e8970d9ebddc02440addf061cb3d44ff4d8d683"}
May 20 14:03:00.965730 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:00.965685    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerStarted","Data":"a7d05f071b68b83d057ed2a4a9b90d59241d5da4787ef8c2c7ea8bea1d7bb1cb"}
May 20 14:03:04.980914 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:04.980874    2557 generic.go:358] "Generic (PLEG): container finished" podID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerID="e7ed56eaad95472f15f56e574e8970d9ebddc02440addf061cb3d44ff4d8d683" exitCode=0
May 20 14:03:04.981304 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:04.980949    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerDied","Data":"e7ed56eaad95472f15f56e574e8970d9ebddc02440addf061cb3d44ff4d8d683"}
May 20 14:03:18.032450 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:18.032397    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerStarted","Data":"250f25d977b48bea4359b66c64e9e217eb426861b448ff3aa29a4c50a96e9f71"}
May 20 14:03:20.039564 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:20.039532    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerStarted","Data":"41b350ce2f0b4888ad9c208074135fa48b82761b9fec6126cfa504aa4ed5a0e5"}
May 20 14:03:23.051701 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:23.051621    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerStarted","Data":"7c4b308935fd6d56ec14770e69e421697d26f137de5f58d1c967468f94b4521f"}
May 20 14:03:23.052084 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:23.051828    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:23.075998 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:23.075950    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podStartSLOduration=5.395677751 podStartE2EDuration="23.075935762s" podCreationTimestamp="2026-05-20 14:03:00 +0000 UTC" firstStartedPulling="2026-05-20 14:03:04.98211298 +0000 UTC m=+765.119667178" lastFinishedPulling="2026-05-20 14:03:22.662370992 +0000 UTC m=+782.799925189" observedRunningTime="2026-05-20 14:03:23.073482273 +0000 UTC m=+783.211036494" watchObservedRunningTime="2026-05-20 14:03:23.075935762 +0000 UTC m=+783.213490296"
May 20 14:03:24.054349 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:24.054307    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:24.054349 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:24.054353    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:24.055784 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:24.055740    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:03:24.056458 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:24.056429    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:24.059260 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:24.059240    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:03:25.057060 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:25.057019    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:03:25.057471 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:25.057326    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:26.059507 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:26.059467    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:03:26.059976 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:26.059728    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:36.060174 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:36.060091    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:03:36.060658 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:36.060633    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:46.059783 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:46.059742    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:03:46.060294 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:46.060258    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:03:56.060217 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:56.060167    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:03:56.060666 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:03:56.060565    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:06.060095 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:06.060052    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:04:06.060502 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:06.060476    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:16.060392 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:16.060344    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:04:16.060804 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:16.060780    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:26.061111 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:26.061076    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:04:26.061601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:26.061160    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:04:35.353206 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.353167    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"]
May 20 14:04:35.353945 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.353769    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" containerID="cri-o://250f25d977b48bea4359b66c64e9e217eb426861b448ff3aa29a4c50a96e9f71" gracePeriod=30
May 20 14:04:35.353945 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.353772    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy" containerID="cri-o://41b350ce2f0b4888ad9c208074135fa48b82761b9fec6126cfa504aa4ed5a0e5" gracePeriod=30
May 20 14:04:35.354157 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.354024    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" containerID="cri-o://7c4b308935fd6d56ec14770e69e421697d26f137de5f58d1c967468f94b4521f" gracePeriod=30
May 20 14:04:35.476504 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.476479    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"]
May 20 14:04:35.479309 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.479290    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.482159 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.482125    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
May 20 14:04:35.482436 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.482418    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
May 20 14:04:35.491505 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.491482    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"]
May 20 14:04:35.667700 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.667633    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906738d7-6b71-4d26-ae62-c862f6763d1a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.667700 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.667674    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.667700 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.667695    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.667918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.667781    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906738d7-6b71-4d26-ae62-c862f6763d1a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.667918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.667814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzpr\" (UniqueName: \"kubernetes.io/projected/906738d7-6b71-4d26-ae62-c862f6763d1a-kube-api-access-8kzpr\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.768549 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.768522    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzpr\" (UniqueName: \"kubernetes.io/projected/906738d7-6b71-4d26-ae62-c862f6763d1a-kube-api-access-8kzpr\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.768676 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.768612    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906738d7-6b71-4d26-ae62-c862f6763d1a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.768754 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.768685    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.768754 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.768716    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.768904 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.768753    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906738d7-6b71-4d26-ae62-c862f6763d1a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.769106 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.769087    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906738d7-6b71-4d26-ae62-c862f6763d1a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.769370 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.769351    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.769417 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.769373    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.771019 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.770998    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906738d7-6b71-4d26-ae62-c862f6763d1a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.776927 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.776910    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzpr\" (UniqueName: \"kubernetes.io/projected/906738d7-6b71-4d26-ae62-c862f6763d1a-kube-api-access-8kzpr\") pod \"isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.789237 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.789219    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:35.918246 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:35.918219    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"]
May 20 14:04:35.920702 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:04:35.920675    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906738d7_6b71_4d26_ae62_c862f6763d1a.slice/crio-a8d413f6886f0927739d69f405fa5c65e260474155eb74e2040b7630f2027d8b WatchSource:0}: Error finding container a8d413f6886f0927739d69f405fa5c65e260474155eb74e2040b7630f2027d8b: Status 404 returned error can't find the container with id a8d413f6886f0927739d69f405fa5c65e260474155eb74e2040b7630f2027d8b
May 20 14:04:36.059869 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:36.059804    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:04:36.060162 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:36.060132    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:36.279532 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:36.279455    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerStarted","Data":"4fa357526814775d702e79debf010f4e86cd1664d63fea69bc957163b057d7b9"}
May 20 14:04:36.279532 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:36.279491    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerStarted","Data":"a8d413f6886f0927739d69f405fa5c65e260474155eb74e2040b7630f2027d8b"}
May 20 14:04:36.281719 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:36.281695    2557 generic.go:358] "Generic (PLEG): container finished" podID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerID="41b350ce2f0b4888ad9c208074135fa48b82761b9fec6126cfa504aa4ed5a0e5" exitCode=2
May 20 14:04:36.281831 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:36.281732    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerDied","Data":"41b350ce2f0b4888ad9c208074135fa48b82761b9fec6126cfa504aa4ed5a0e5"}
May 20 14:04:39.055170 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:39.055131    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
May 20 14:04:39.296717 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:39.296635    2557 generic.go:358] "Generic (PLEG): container finished" podID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerID="250f25d977b48bea4359b66c64e9e217eb426861b448ff3aa29a4c50a96e9f71" exitCode=0
May 20 14:04:39.296841 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:39.296716    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerDied","Data":"250f25d977b48bea4359b66c64e9e217eb426861b448ff3aa29a4c50a96e9f71"}
May 20 14:04:39.819141 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:04:39.819111    2557 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906738d7_6b71_4d26_ae62_c862f6763d1a.slice/crio-4fa357526814775d702e79debf010f4e86cd1664d63fea69bc957163b057d7b9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906738d7_6b71_4d26_ae62_c862f6763d1a.slice/crio-conmon-4fa357526814775d702e79debf010f4e86cd1664d63fea69bc957163b057d7b9.scope\": RecentStats: unable to find data in memory cache]"
May 20 14:04:40.300707 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:40.300678    2557 generic.go:358] "Generic (PLEG): container finished" podID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerID="4fa357526814775d702e79debf010f4e86cd1664d63fea69bc957163b057d7b9" exitCode=0
May 20 14:04:40.301135 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:40.300722    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerDied","Data":"4fa357526814775d702e79debf010f4e86cd1664d63fea69bc957163b057d7b9"}
May 20 14:04:41.306103 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:41.306070    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerStarted","Data":"6495637a36960cfa603378844d69f53c0166442be9d10e0e35c7f5eda6c93bb5"}
May 20 14:04:41.306103 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:41.306103    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerStarted","Data":"f7d5a9e22e07d5942dadf7847876a5c05ccb98540697362685a09b7230e11d9f"}
May 20 14:04:41.306501 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:41.306112    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerStarted","Data":"401849cb70a68b29f47b77e26a181ccab1bf3e0737bedff7e6c8153eb7011000"}
May 20 14:04:41.306501 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:41.306406    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:41.306596 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:41.306539    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:41.307677 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:41.307652    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:04:41.329652 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:41.329607    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podStartSLOduration=6.329594458 podStartE2EDuration="6.329594458s" podCreationTimestamp="2026-05-20 14:04:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:04:41.328156486 +0000 UTC m=+861.465710717" watchObservedRunningTime="2026-05-20 14:04:41.329594458 +0000 UTC m=+861.467148673"
May 20 14:04:42.309743 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:42.309704    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:42.310227 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:42.309886    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:04:42.310746 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:42.310724    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:43.312679 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:43.312645    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:04:43.313084 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:43.313000    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:44.055303 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:44.055269    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
May 20 14:04:46.060304 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:46.060263    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:04:46.060701 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:46.060579    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:48.317429 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:48.317401    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:04:48.317967 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:48.317939    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:04:48.318231 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:48.318207    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:49.055285 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:49.055243    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
May 20 14:04:49.055460 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:49.055383    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:04:54.055124 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:54.055084    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
May 20 14:04:56.060132 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:56.060087    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 20 14:04:56.060543 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:56.060268    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:04:56.060543 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:56.060425    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:56.060543 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:56.060522    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:04:58.318080 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:58.318039    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:04:58.318583 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:58.318521    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:04:59.055112 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:04:59.055070    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
May 20 14:05:04.055177 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:04.055085    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
May 20 14:05:05.387541 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.387493    2557 generic.go:358] "Generic (PLEG): container finished" podID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerID="7c4b308935fd6d56ec14770e69e421697d26f137de5f58d1c967468f94b4521f" exitCode=0
May 20 14:05:05.387962 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.387564    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerDied","Data":"7c4b308935fd6d56ec14770e69e421697d26f137de5f58d1c967468f94b4521f"}
May 20 14:05:05.508119 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.508099    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:05:05.610906 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.610834    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") "
May 20 14:05:05.611024 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.610908    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kserve-provision-location\") pod \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") "
May 20 14:05:05.611024 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.610954    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-proxy-tls\") pod \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") "
May 20 14:05:05.611024 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.610982    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzv9k\" (UniqueName: \"kubernetes.io/projected/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kube-api-access-kzv9k\") pod \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") "
May 20 14:05:05.611191 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.611039    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-cabundle-cert\") pod \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\" (UID: \"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f\") "
May 20 14:05:05.611257 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.611208    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" (UID: "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:05:05.611257 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.611217    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" (UID: "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:05:05.611372 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.611344    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:05:05.611372 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.611366    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:05:05.611468 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.611427    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" (UID: "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:05:05.613142 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.613114    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" (UID: "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:05:05.613231 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.613155    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kube-api-access-kzv9k" (OuterVolumeSpecName: "kube-api-access-kzv9k") pod "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" (UID: "0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f"). InnerVolumeSpecName "kube-api-access-kzv9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:05:05.712348 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.712325    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:05:05.712348 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.712348    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzv9k\" (UniqueName: \"kubernetes.io/projected/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-kube-api-access-kzv9k\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:05:05.712504 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:05.712358    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:05:06.393163 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:06.393131    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt" event={"ID":"0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f","Type":"ContainerDied","Data":"a7d05f071b68b83d057ed2a4a9b90d59241d5da4787ef8c2c7ea8bea1d7bb1cb"}
May 20 14:05:06.393606 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:06.393182    2557 scope.go:117] "RemoveContainer" containerID="7c4b308935fd6d56ec14770e69e421697d26f137de5f58d1c967468f94b4521f"
May 20 14:05:06.393606 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:06.393183    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"
May 20 14:05:06.401338 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:06.401320    2557 scope.go:117] "RemoveContainer" containerID="41b350ce2f0b4888ad9c208074135fa48b82761b9fec6126cfa504aa4ed5a0e5"
May 20 14:05:06.409028 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:06.409013    2557 scope.go:117] "RemoveContainer" containerID="250f25d977b48bea4359b66c64e9e217eb426861b448ff3aa29a4c50a96e9f71"
May 20 14:05:06.416703 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:06.416681    2557 scope.go:117] "RemoveContainer" containerID="e7ed56eaad95472f15f56e574e8970d9ebddc02440addf061cb3d44ff4d8d683"
May 20 14:05:06.416976 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:06.416956    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"]
May 20 14:05:06.421908 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:06.421887    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6ffbc8fbd7-vqgzt"]
May 20 14:05:08.318123 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:08.318078    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:05:08.318586 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:08.318563    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:05:08.405096 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:08.405064    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" path="/var/lib/kubelet/pods/0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f/volumes"
May 20 14:05:18.318477 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:18.318437    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:05:18.318918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:18.318897    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:05:20.345475 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:20.345445    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:05:20.347424 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:20.347405    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:05:20.349789 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:20.349766    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:05:20.352156 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:20.352137    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:05:28.317941 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:28.317902    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:05:28.318491 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:28.318456    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:05:38.318644 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:38.318598    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:05:38.319120 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:38.319076    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:05:48.319062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:48.319025    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:05:48.319465 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:05:48.319165    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:06:00.537251 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:00.537213    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"]
May 20 14:06:00.537793 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:00.537555    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" containerID="cri-o://401849cb70a68b29f47b77e26a181ccab1bf3e0737bedff7e6c8153eb7011000" gracePeriod=30
May 20 14:06:00.537793 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:00.537584    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" containerID="cri-o://6495637a36960cfa603378844d69f53c0166442be9d10e0e35c7f5eda6c93bb5" gracePeriod=30
May 20 14:06:00.537793 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:00.537606    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy" containerID="cri-o://f7d5a9e22e07d5942dadf7847876a5c05ccb98540697362685a09b7230e11d9f" gracePeriod=30
May 20 14:06:01.575235 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:01.575204    2557 generic.go:358] "Generic (PLEG): container finished" podID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerID="f7d5a9e22e07d5942dadf7847876a5c05ccb98540697362685a09b7230e11d9f" exitCode=2
May 20 14:06:01.575607 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:01.575243    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerDied","Data":"f7d5a9e22e07d5942dadf7847876a5c05ccb98540697362685a09b7230e11d9f"}
May 20 14:06:03.313022 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:03.312975    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 14:06:04.586414 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:04.586382    2557 generic.go:358] "Generic (PLEG): container finished" podID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerID="401849cb70a68b29f47b77e26a181ccab1bf3e0737bedff7e6c8153eb7011000" exitCode=0
May 20 14:06:04.586788 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:04.586461    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerDied","Data":"401849cb70a68b29f47b77e26a181ccab1bf3e0737bedff7e6c8153eb7011000"}
May 20 14:06:08.312917 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:08.312834    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 14:06:08.318265 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:08.318214    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:06:08.318514 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:08.318482    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:06:13.313240 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:13.313198    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 14:06:13.313619 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:13.313332    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:06:18.313622 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:18.313578    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 14:06:18.317952 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:18.317918    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:06:18.318302 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:18.318281    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:06:20.675943 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.675909    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"]
May 20 14:06:20.676494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676460    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="storage-initializer"
May 20 14:06:20.676494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676482    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="storage-initializer"
May 20 14:06:20.676670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676497    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container"
May 20 14:06:20.676670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676506    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container"
May 20 14:06:20.676670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676522    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent"
May 20 14:06:20.676670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676529    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent"
May 20 14:06:20.676670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676559    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy"
May 20 14:06:20.676670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676567    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy"
May 20 14:06:20.676670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676668    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kserve-container"
May 20 14:06:20.677033 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676685    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="kube-rbac-proxy"
May 20 14:06:20.677033 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.676697    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="0b5c4776-7e80-4fe4-9700-1e6ba0f41c7f" containerName="agent"
May 20 14:06:20.680228 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.680206    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.682937 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.682912    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
May 20 14:06:20.683041 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.682956    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
May 20 14:06:20.690802 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.690777    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"]
May 20 14:06:20.792065 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.792039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hr2x\" (UniqueName: \"kubernetes.io/projected/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kube-api-access-2hr2x\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.792185 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.792083    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-proxy-tls\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.792185 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.792101    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-cabundle-cert\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.792185 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.792175    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kserve-provision-location\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.792306 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.792195    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.892715 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.892686    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kserve-provision-location\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.892832 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.892728    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.892832 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.892772    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2hr2x\" (UniqueName: \"kubernetes.io/projected/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kube-api-access-2hr2x\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.892832 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.892802    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-proxy-tls\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.892832 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.892818    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-cabundle-cert\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.893626 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.893599    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kserve-provision-location\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.893895 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.893787    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-cabundle-cert\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.894171 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.894148    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.901453 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.901427    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-proxy-tls\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.901994 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.901976    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hr2x\" (UniqueName: \"kubernetes.io/projected/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kube-api-access-2hr2x\") pod \"isvc-logger-predictor-6899674c9d-l89rx\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:20.991394 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:20.991339    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:21.119656 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:21.119629    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"]
May 20 14:06:21.121354 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:06:21.121327    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8ce51e_836f_47b2_9cdc_039d2090c3cb.slice/crio-b9eea5bee7a82418d02d0bd44b94e046fa4ff0b070e0a7101bf20c57705444b8 WatchSource:0}: Error finding container b9eea5bee7a82418d02d0bd44b94e046fa4ff0b070e0a7101bf20c57705444b8: Status 404 returned error can't find the container with id b9eea5bee7a82418d02d0bd44b94e046fa4ff0b070e0a7101bf20c57705444b8
May 20 14:06:21.123453 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:21.123436    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:06:21.644175 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:21.644139    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerStarted","Data":"5f98b90c30bf6e6780e03d1282b436f87c68eb5d5ecbcf67a521590e972c6b81"}
May 20 14:06:21.644337 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:21.644180    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerStarted","Data":"b9eea5bee7a82418d02d0bd44b94e046fa4ff0b070e0a7101bf20c57705444b8"}
May 20 14:06:23.312788 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:23.312739    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 14:06:25.659453 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:25.659416    2557 generic.go:358] "Generic (PLEG): container finished" podID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerID="5f98b90c30bf6e6780e03d1282b436f87c68eb5d5ecbcf67a521590e972c6b81" exitCode=0
May 20 14:06:25.659918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:25.659481    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerDied","Data":"5f98b90c30bf6e6780e03d1282b436f87c68eb5d5ecbcf67a521590e972c6b81"}
May 20 14:06:26.664818 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:26.664787    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerStarted","Data":"9f61183a14cfe369ac38591bbca3afc562a08bb98d5cbeebeba3802fda73a4d8"}
May 20 14:06:26.664818 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:26.664823    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerStarted","Data":"10364ea9d8337f589c9faa30944bf7a7f29b07196e285876a58710838721d65a"}
May 20 14:06:26.664818 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:26.664833    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerStarted","Data":"755e680cb44e0b6626011ae64a8238aecc6d1350e41aa3598f77c475f5885f15"}
May 20 14:06:26.665289 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:26.665207    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:26.665342 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:26.665297    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:26.666530 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:26.666503    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:06:26.689224 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:26.689170    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podStartSLOduration=6.689155185 podStartE2EDuration="6.689155185s" podCreationTimestamp="2026-05-20 14:06:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:06:26.687521508 +0000 UTC m=+966.825075727" watchObservedRunningTime="2026-05-20 14:06:26.689155185 +0000 UTC m=+966.826709404"
May 20 14:06:27.671157 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:27.671107    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:06:27.671572 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:27.671139    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:27.672020 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:27.671995    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:06:28.313670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:28.313631    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
May 20 14:06:28.318066 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:28.318037    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:5000: connect: connection refused"
May 20 14:06:28.318188 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:28.318173    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:06:28.318320 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:28.318296    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:06:28.318409 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:28.318395    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:06:28.674084 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:28.674050    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:06:28.674564 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:28.674533    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:06:30.682883 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.682806    2557 generic.go:358] "Generic (PLEG): container finished" podID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerID="6495637a36960cfa603378844d69f53c0166442be9d10e0e35c7f5eda6c93bb5" exitCode=0
May 20 14:06:30.682883 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.682843    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerDied","Data":"6495637a36960cfa603378844d69f53c0166442be9d10e0e35c7f5eda6c93bb5"}
May 20 14:06:30.683193 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.682881    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q" event={"ID":"906738d7-6b71-4d26-ae62-c862f6763d1a","Type":"ContainerDied","Data":"a8d413f6886f0927739d69f405fa5c65e260474155eb74e2040b7630f2027d8b"}
May 20 14:06:30.683193 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.682895    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8d413f6886f0927739d69f405fa5c65e260474155eb74e2040b7630f2027d8b"
May 20 14:06:30.689907 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.689888    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:06:30.769398 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.769370    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906738d7-6b71-4d26-ae62-c862f6763d1a-kserve-provision-location\") pod \"906738d7-6b71-4d26-ae62-c862f6763d1a\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") "
May 20 14:06:30.769536 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.769423    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kzpr\" (UniqueName: \"kubernetes.io/projected/906738d7-6b71-4d26-ae62-c862f6763d1a-kube-api-access-8kzpr\") pod \"906738d7-6b71-4d26-ae62-c862f6763d1a\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") "
May 20 14:06:30.769536 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.769474    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-cabundle-cert\") pod \"906738d7-6b71-4d26-ae62-c862f6763d1a\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") "
May 20 14:06:30.769536 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.769497    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906738d7-6b71-4d26-ae62-c862f6763d1a-proxy-tls\") pod \"906738d7-6b71-4d26-ae62-c862f6763d1a\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") "
May 20 14:06:30.769694 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.769569    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"906738d7-6b71-4d26-ae62-c862f6763d1a\" (UID: \"906738d7-6b71-4d26-ae62-c862f6763d1a\") "
May 20 14:06:30.769774 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.769752    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906738d7-6b71-4d26-ae62-c862f6763d1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "906738d7-6b71-4d26-ae62-c862f6763d1a" (UID: "906738d7-6b71-4d26-ae62-c862f6763d1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:06:30.769829 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.769820    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906738d7-6b71-4d26-ae62-c862f6763d1a-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:06:30.769909 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.769823    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "906738d7-6b71-4d26-ae62-c862f6763d1a" (UID: "906738d7-6b71-4d26-ae62-c862f6763d1a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:06:30.770043 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.770006    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "906738d7-6b71-4d26-ae62-c862f6763d1a" (UID: "906738d7-6b71-4d26-ae62-c862f6763d1a"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:06:30.771619 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.771594    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906738d7-6b71-4d26-ae62-c862f6763d1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "906738d7-6b71-4d26-ae62-c862f6763d1a" (UID: "906738d7-6b71-4d26-ae62-c862f6763d1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:06:30.771696 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.771595    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906738d7-6b71-4d26-ae62-c862f6763d1a-kube-api-access-8kzpr" (OuterVolumeSpecName: "kube-api-access-8kzpr") pod "906738d7-6b71-4d26-ae62-c862f6763d1a" (UID: "906738d7-6b71-4d26-ae62-c862f6763d1a"). InnerVolumeSpecName "kube-api-access-8kzpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:06:30.870780 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.870760    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:06:30.870900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.870783    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kzpr\" (UniqueName: \"kubernetes.io/projected/906738d7-6b71-4d26-ae62-c862f6763d1a-kube-api-access-8kzpr\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:06:30.870900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.870794    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906738d7-6b71-4d26-ae62-c862f6763d1a-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:06:30.870900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:30.870803    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906738d7-6b71-4d26-ae62-c862f6763d1a-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:06:31.686406 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:31.686376    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"
May 20 14:06:31.709768 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:31.709737    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"]
May 20 14:06:31.713493 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:31.713470    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-d7c47fb6b-x8k8q"]
May 20 14:06:32.404839 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:32.404810    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" path="/var/lib/kubelet/pods/906738d7-6b71-4d26-ae62-c862f6763d1a/volumes"
May 20 14:06:33.678021 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:33.677992    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:06:33.678587 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:33.678558    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:06:33.679125 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:33.679098    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:06:43.678614 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:43.678561    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:06:43.679140 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:43.679108    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:06:53.678645 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:53.678599    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:06:53.679090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:06:53.679066    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:07:03.678604 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:03.678559    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:07:03.679067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:03.678961    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:07:13.678547 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:13.678502    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:07:13.679090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:13.679066    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:07:23.678656 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:23.678608    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:07:23.679175 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:23.679109    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:07:33.679511 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:33.679471    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:07:33.679914 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:33.679600    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:07:45.894088 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.894021    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"]
May 20 14:07:45.894585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.894399    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" containerID="cri-o://755e680cb44e0b6626011ae64a8238aecc6d1350e41aa3598f77c475f5885f15" gracePeriod=30
May 20 14:07:45.894585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.894482    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy" containerID="cri-o://10364ea9d8337f589c9faa30944bf7a7f29b07196e285876a58710838721d65a" gracePeriod=30
May 20 14:07:45.894585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.894406    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" containerID="cri-o://9f61183a14cfe369ac38591bbca3afc562a08bb98d5cbeebeba3802fda73a4d8" gracePeriod=30
May 20 14:07:45.961308 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961282    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"]
May 20 14:07:45.961634 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961623    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy"
May 20 14:07:45.961678 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961636    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy"
May 20 14:07:45.961678 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961652    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container"
May 20 14:07:45.961678 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961658    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container"
May 20 14:07:45.961678 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961668    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="storage-initializer"
May 20 14:07:45.961678 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961674    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="storage-initializer"
May 20 14:07:45.961829 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961685    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent"
May 20 14:07:45.961829 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961690    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent"
May 20 14:07:45.961829 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961746    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="agent"
May 20 14:07:45.961829 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961758    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kserve-container"
May 20 14:07:45.961829 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.961765    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="906738d7-6b71-4d26-ae62-c862f6763d1a" containerName="kube-rbac-proxy"
May 20 14:07:45.965090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.965074    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:45.967870 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.967843    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
May 20 14:07:45.967949 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.967913    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
May 20 14:07:45.976810 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:45.976788    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"]
May 20 14:07:46.032813 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.032787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.032936 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.032829    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.032936 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.032875    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-cabundle-cert\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.032936 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.032917    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ct2m\" (UniqueName: \"kubernetes.io/projected/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kube-api-access-5ct2m\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.033113 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.032988    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.134193 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.134167    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.134293 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.134210    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.134293 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.134232    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.134405 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.134345    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-cabundle-cert\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.134459 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.134405    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5ct2m\" (UniqueName: \"kubernetes.io/projected/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kube-api-access-5ct2m\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.134613 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.134588    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.135029 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.135007    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-cabundle-cert\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.135125 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.135088    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.136486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.136470    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.143539 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.143520    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ct2m\" (UniqueName: \"kubernetes.io/projected/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kube-api-access-5ct2m\") pod \"isvc-lightgbm-predictor-bdf964bd-9mdp4\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.274318 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.274268    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:46.408817 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.408795    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"]
May 20 14:07:46.411195 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:07:46.411167    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5113ae0_dfae_418e_a047_ddd6b83b3eb0.slice/crio-e3d67f1081d9bdd770ab40c54c76352527e8394b2e45a0845b7976c7f2e86ee6 WatchSource:0}: Error finding container e3d67f1081d9bdd770ab40c54c76352527e8394b2e45a0845b7976c7f2e86ee6: Status 404 returned error can't find the container with id e3d67f1081d9bdd770ab40c54c76352527e8394b2e45a0845b7976c7f2e86ee6
May 20 14:07:46.927781 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.927748    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" event={"ID":"d5113ae0-dfae-418e-a047-ddd6b83b3eb0","Type":"ContainerStarted","Data":"0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339"}
May 20 14:07:46.928238 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.927786    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" event={"ID":"d5113ae0-dfae-418e-a047-ddd6b83b3eb0","Type":"ContainerStarted","Data":"e3d67f1081d9bdd770ab40c54c76352527e8394b2e45a0845b7976c7f2e86ee6"}
May 20 14:07:46.930105 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.930078    2557 generic.go:358] "Generic (PLEG): container finished" podID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerID="10364ea9d8337f589c9faa30944bf7a7f29b07196e285876a58710838721d65a" exitCode=2
May 20 14:07:46.930230 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:46.930127    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerDied","Data":"10364ea9d8337f589c9faa30944bf7a7f29b07196e285876a58710838721d65a"}
May 20 14:07:48.674560 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:48.674516    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 20 14:07:49.943195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:49.943169    2557 generic.go:358] "Generic (PLEG): container finished" podID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerID="755e680cb44e0b6626011ae64a8238aecc6d1350e41aa3598f77c475f5885f15" exitCode=0
May 20 14:07:49.943519 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:49.943249    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerDied","Data":"755e680cb44e0b6626011ae64a8238aecc6d1350e41aa3598f77c475f5885f15"}
May 20 14:07:50.950259 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:50.950217    2557 generic.go:358] "Generic (PLEG): container finished" podID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerID="0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339" exitCode=0
May 20 14:07:50.950673 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:50.950292    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" event={"ID":"d5113ae0-dfae-418e-a047-ddd6b83b3eb0","Type":"ContainerDied","Data":"0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339"}
May 20 14:07:53.675136 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:53.675087    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 20 14:07:53.679456 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:53.679413    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:07:53.679777 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:53.679746    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:07:57.977898 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:57.977841    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" event={"ID":"d5113ae0-dfae-418e-a047-ddd6b83b3eb0","Type":"ContainerStarted","Data":"4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191"}
May 20 14:07:57.977898 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:57.977906    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" event={"ID":"d5113ae0-dfae-418e-a047-ddd6b83b3eb0","Type":"ContainerStarted","Data":"b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950"}
May 20 14:07:57.978330 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:57.978108    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:58.002307 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:58.002261    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podStartSLOduration=6.648911904 podStartE2EDuration="13.002245299s" podCreationTimestamp="2026-05-20 14:07:45 +0000 UTC" firstStartedPulling="2026-05-20 14:07:50.951605925 +0000 UTC m=+1051.089160122" lastFinishedPulling="2026-05-20 14:07:57.304939301 +0000 UTC m=+1057.442493517" observedRunningTime="2026-05-20 14:07:57.999951691 +0000 UTC m=+1058.137505911" watchObservedRunningTime="2026-05-20 14:07:58.002245299 +0000 UTC m=+1058.139799520"
May 20 14:07:58.675193 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:58.675150    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 20 14:07:58.675376 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:58.675268    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:07:58.981870 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:58.981768    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:07:58.983036 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:58.983009    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:07:59.984924 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:07:59.984884    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:08:03.675286 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:03.675204    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 20 14:08:03.678545 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:03.678517    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:08:03.678812 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:03.678788    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:08:04.990001 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:04.989968    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:08:04.990591 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:04.990565    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:08:08.674629 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:08.674584    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 20 14:08:13.675334 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:13.675282    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 20 14:08:13.678781 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:13.678748    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 20 14:08:13.678947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:13.678929    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:08:13.679180 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:13.679151    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 14:08:13.679306 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:13.679291    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:08:14.990686 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:14.990649    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:08:16.044254 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.044227    2557 generic.go:358] "Generic (PLEG): container finished" podID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerID="9f61183a14cfe369ac38591bbca3afc562a08bb98d5cbeebeba3802fda73a4d8" exitCode=0
May 20 14:08:16.044644 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.044271    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerDied","Data":"9f61183a14cfe369ac38591bbca3afc562a08bb98d5cbeebeba3802fda73a4d8"}
May 20 14:08:16.044644 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.044299    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx" event={"ID":"5c8ce51e-836f-47b2-9cdc-039d2090c3cb","Type":"ContainerDied","Data":"b9eea5bee7a82418d02d0bd44b94e046fa4ff0b070e0a7101bf20c57705444b8"}
May 20 14:08:16.044644 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.044315    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9eea5bee7a82418d02d0bd44b94e046fa4ff0b070e0a7101bf20c57705444b8"
May 20 14:08:16.050196 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.050165    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:08:16.173848 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.173770    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-isvc-logger-kube-rbac-proxy-sar-config\") pod \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") "
May 20 14:08:16.173848 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.173806    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-cabundle-cert\") pod \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") "
May 20 14:08:16.174090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.173884    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kserve-provision-location\") pod \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") "
May 20 14:08:16.174090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.173923    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-proxy-tls\") pod \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") "
May 20 14:08:16.174090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.173943    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hr2x\" (UniqueName: \"kubernetes.io/projected/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kube-api-access-2hr2x\") pod \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\" (UID: \"5c8ce51e-836f-47b2-9cdc-039d2090c3cb\") "
May 20 14:08:16.174253 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.174183    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "5c8ce51e-836f-47b2-9cdc-039d2090c3cb" (UID: "5c8ce51e-836f-47b2-9cdc-039d2090c3cb"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:08:16.174253 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.174196    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c8ce51e-836f-47b2-9cdc-039d2090c3cb" (UID: "5c8ce51e-836f-47b2-9cdc-039d2090c3cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:08:16.174253 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.174224    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c8ce51e-836f-47b2-9cdc-039d2090c3cb" (UID: "5c8ce51e-836f-47b2-9cdc-039d2090c3cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:08:16.175963 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.175940    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kube-api-access-2hr2x" (OuterVolumeSpecName: "kube-api-access-2hr2x") pod "5c8ce51e-836f-47b2-9cdc-039d2090c3cb" (UID: "5c8ce51e-836f-47b2-9cdc-039d2090c3cb"). InnerVolumeSpecName "kube-api-access-2hr2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:08:16.176060 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.176032    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c8ce51e-836f-47b2-9cdc-039d2090c3cb" (UID: "5c8ce51e-836f-47b2-9cdc-039d2090c3cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:08:16.274911 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.274878    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hr2x\" (UniqueName: \"kubernetes.io/projected/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kube-api-access-2hr2x\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:08:16.274911 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.274909    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:08:16.274911 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.274922    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:08:16.275322 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.274931    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:08:16.275322 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:16.274942    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c8ce51e-836f-47b2-9cdc-039d2090c3cb-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:08:17.047468 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:17.047437    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"
May 20 14:08:17.076715 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:17.076680    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"]
May 20 14:08:17.080163 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:17.080135    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6899674c9d-l89rx"]
May 20 14:08:18.405156 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:18.405123    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" path="/var/lib/kubelet/pods/5c8ce51e-836f-47b2-9cdc-039d2090c3cb/volumes"
May 20 14:08:24.991079 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:24.991031    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:08:34.990600 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:34.990562    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:08:44.991075 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:44.991036    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:08:54.990578 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:08:54.990540    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:09:04.990954 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:04.990909    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:09:05.401332 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:05.401295    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 20 14:09:15.402114 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:15.402072    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:09:16.078305 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.078273    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"]
May 20 14:09:16.226280 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226244    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"]
May 20 14:09:16.226601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226588    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="storage-initializer"
May 20 14:09:16.226643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226603    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="storage-initializer"
May 20 14:09:16.226643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226611    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent"
May 20 14:09:16.226643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226616    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent"
May 20 14:09:16.226643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226623    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy"
May 20 14:09:16.226643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226630    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy"
May 20 14:09:16.226643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226638    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container"
May 20 14:09:16.226643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226643    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container"
May 20 14:09:16.226894 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226706    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kube-rbac-proxy"
May 20 14:09:16.226894 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226721    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="kserve-container"
May 20 14:09:16.226894 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.226733    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c8ce51e-836f-47b2-9cdc-039d2090c3cb" containerName="agent"
May 20 14:09:16.229076 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.229059    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.232009 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.231983    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
May 20 14:09:16.232009 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.231983    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
May 20 14:09:16.242400 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.242381    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"]
May 20 14:09:16.254027 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.253978    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container" containerID="cri-o://b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950" gracePeriod=30
May 20 14:09:16.254127 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.254063    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kube-rbac-proxy" containerID="cri-o://4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191" gracePeriod=30
May 20 14:09:16.322075 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.322055    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.322215 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.322089    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.322215 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.322117    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.322215 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.322172    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.322215 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.322195    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqrqp\" (UniqueName: \"kubernetes.io/projected/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kube-api-access-bqrqp\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.423143 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.423121    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.423446 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.423163    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.423446 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.423190    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.423446 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.423231    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.423446 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.423270    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bqrqp\" (UniqueName: \"kubernetes.io/projected/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kube-api-access-bqrqp\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.423446 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:09:16.423387    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 20 14:09:16.423714 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:09:16.423459    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-proxy-tls podName:b0f3fb95-2fd7-487a-b619-2f813d3f79e9 nodeName:}" failed. No retries permitted until 2026-05-20 14:09:16.923438953 +0000 UTC m=+1137.060993157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-proxy-tls") pod "isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" (UID: "b0f3fb95-2fd7-487a-b619-2f813d3f79e9") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 20 14:09:16.423714 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.423646    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.423797 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.423757    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.423886 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.423846    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.433087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.433067    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqrqp\" (UniqueName: \"kubernetes.io/projected/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kube-api-access-bqrqp\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.927694 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.927662    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:16.929915 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:16.929886    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:17.138915 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:17.138882    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:17.262975 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:17.262942    2557 generic.go:358] "Generic (PLEG): container finished" podID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerID="4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191" exitCode=2
May 20 14:09:17.263099 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:17.263005    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" event={"ID":"d5113ae0-dfae-418e-a047-ddd6b83b3eb0","Type":"ContainerDied","Data":"4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191"}
May 20 14:09:17.269488 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:17.269456    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"]
May 20 14:09:17.272424 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:09:17.272393    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f3fb95_2fd7_487a_b619_2f813d3f79e9.slice/crio-8405663156e53f9ba61ca988032f17e77c089ac8ea2a59b43077f81fca4dd9be WatchSource:0}: Error finding container 8405663156e53f9ba61ca988032f17e77c089ac8ea2a59b43077f81fca4dd9be: Status 404 returned error can't find the container with id 8405663156e53f9ba61ca988032f17e77c089ac8ea2a59b43077f81fca4dd9be
May 20 14:09:18.268174 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:18.268141    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" event={"ID":"b0f3fb95-2fd7-487a-b619-2f813d3f79e9","Type":"ContainerStarted","Data":"7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed"}
May 20 14:09:18.268552 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:18.268186    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" event={"ID":"b0f3fb95-2fd7-487a-b619-2f813d3f79e9","Type":"ContainerStarted","Data":"8405663156e53f9ba61ca988032f17e77c089ac8ea2a59b43077f81fca4dd9be"}
May 20 14:09:19.994615 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:19.994588    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:09:20.153116 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.153091    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") "
May 20 14:09:20.153277 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.153162    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ct2m\" (UniqueName: \"kubernetes.io/projected/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kube-api-access-5ct2m\") pod \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") "
May 20 14:09:20.153277 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.153259    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-cabundle-cert\") pod \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") "
May 20 14:09:20.153398 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.153293    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-proxy-tls\") pod \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") "
May 20 14:09:20.153398 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.153314    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kserve-provision-location\") pod \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\" (UID: \"d5113ae0-dfae-418e-a047-ddd6b83b3eb0\") "
May 20 14:09:20.153538 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.153510    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "d5113ae0-dfae-418e-a047-ddd6b83b3eb0" (UID: "d5113ae0-dfae-418e-a047-ddd6b83b3eb0"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:09:20.153648 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.153575    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5113ae0-dfae-418e-a047-ddd6b83b3eb0" (UID: "d5113ae0-dfae-418e-a047-ddd6b83b3eb0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:09:20.153781 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.153674    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5113ae0-dfae-418e-a047-ddd6b83b3eb0" (UID: "d5113ae0-dfae-418e-a047-ddd6b83b3eb0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:20.155262 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.155241    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kube-api-access-5ct2m" (OuterVolumeSpecName: "kube-api-access-5ct2m") pod "d5113ae0-dfae-418e-a047-ddd6b83b3eb0" (UID: "d5113ae0-dfae-418e-a047-ddd6b83b3eb0"). InnerVolumeSpecName "kube-api-access-5ct2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:09:20.155360 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.155241    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5113ae0-dfae-418e-a047-ddd6b83b3eb0" (UID: "d5113ae0-dfae-418e-a047-ddd6b83b3eb0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:09:20.254300 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.254277    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ct2m\" (UniqueName: \"kubernetes.io/projected/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kube-api-access-5ct2m\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:09:20.254300 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.254298    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:09:20.254418 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.254308    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:09:20.254418 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.254317    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:09:20.254418 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.254327    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5113ae0-dfae-418e-a047-ddd6b83b3eb0-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:09:20.276445 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.276417    2557 generic.go:358] "Generic (PLEG): container finished" podID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerID="b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950" exitCode=0
May 20 14:09:20.276527 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.276504    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" event={"ID":"d5113ae0-dfae-418e-a047-ddd6b83b3eb0","Type":"ContainerDied","Data":"b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950"}
May 20 14:09:20.276567 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.276540    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" event={"ID":"d5113ae0-dfae-418e-a047-ddd6b83b3eb0","Type":"ContainerDied","Data":"e3d67f1081d9bdd770ab40c54c76352527e8394b2e45a0845b7976c7f2e86ee6"}
May 20 14:09:20.276567 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.276557    2557 scope.go:117] "RemoveContainer" containerID="4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191"
May 20 14:09:20.276627 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.276512    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"
May 20 14:09:20.285897 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.285703    2557 scope.go:117] "RemoveContainer" containerID="b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950"
May 20 14:09:20.295347 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.295331    2557 scope.go:117] "RemoveContainer" containerID="0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339"
May 20 14:09:20.302286 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.302261    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"]
May 20 14:09:20.302653 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.302639    2557 scope.go:117] "RemoveContainer" containerID="4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191"
May 20 14:09:20.302949 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:09:20.302928    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191\": container with ID starting with 4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191 not found: ID does not exist" containerID="4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191"
May 20 14:09:20.303011 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.302961    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191"} err="failed to get container status \"4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191\": rpc error: code = NotFound desc = could not find container \"4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191\": container with ID starting with 4152897c318b1004d536830ec6a243eb7bb3a62dc977572b3743dd8a4dbac191 not found: ID does not exist"
May 20 14:09:20.303011 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.302985    2557 scope.go:117] "RemoveContainer" containerID="b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950"
May 20 14:09:20.303220 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:09:20.303199    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950\": container with ID starting with b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950 not found: ID does not exist" containerID="b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950"
May 20 14:09:20.303268 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.303226    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950"} err="failed to get container status \"b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950\": rpc error: code = NotFound desc = could not find container \"b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950\": container with ID starting with b068dbead20abc71745ce8314f79a7bd2729a22fcc36957a2cd1a8bb54dfc950 not found: ID does not exist"
May 20 14:09:20.303268 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.303244    2557 scope.go:117] "RemoveContainer" containerID="0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339"
May 20 14:09:20.303501 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:09:20.303485    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339\": container with ID starting with 0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339 not found: ID does not exist" containerID="0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339"
May 20 14:09:20.303538 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.303505    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339"} err="failed to get container status \"0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339\": rpc error: code = NotFound desc = could not find container \"0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339\": container with ID starting with 0e04e01f05de89191ae5014ce2d43918d71227bc9658422c949b83c15e212339 not found: ID does not exist"
May 20 14:09:20.306656 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.306636    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4"]
May 20 14:09:20.404615 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.404565    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" path="/var/lib/kubelet/pods/d5113ae0-dfae-418e-a047-ddd6b83b3eb0/volumes"
May 20 14:09:20.986183 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:20.986140    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-9mdp4" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
May 20 14:09:21.281499 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:21.281434    2557 generic.go:358] "Generic (PLEG): container finished" podID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerID="7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed" exitCode=0
May 20 14:09:21.281499 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:21.281485    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" event={"ID":"b0f3fb95-2fd7-487a-b619-2f813d3f79e9","Type":"ContainerDied","Data":"7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed"}
May 20 14:09:22.287045 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:22.287008    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" event={"ID":"b0f3fb95-2fd7-487a-b619-2f813d3f79e9","Type":"ContainerStarted","Data":"56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379"}
May 20 14:09:22.287045 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:22.287049    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" event={"ID":"b0f3fb95-2fd7-487a-b619-2f813d3f79e9","Type":"ContainerStarted","Data":"241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27"}
May 20 14:09:22.287516 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:22.287250    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:22.314397 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:22.314357    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podStartSLOduration=6.314344392 podStartE2EDuration="6.314344392s" podCreationTimestamp="2026-05-20 14:09:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:09:22.312558255 +0000 UTC m=+1142.450112474" watchObservedRunningTime="2026-05-20 14:09:22.314344392 +0000 UTC m=+1142.451898666"
May 20 14:09:23.290691 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:23.290651    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:23.291834 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:23.291793    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:09:24.293949 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:24.293905    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:09:29.298429 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:29.298399    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:09:29.299038 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:29.299006    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:09:39.299393 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:39.299310    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:09:49.299300 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:49.299259    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:09:59.299479 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:09:59.299440    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:10:09.299654 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:09.299613    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:10:19.299128 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:19.299088    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:10:20.374061 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:20.374029    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:10:20.377168 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:20.377145    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:10:20.378239 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:20.378212    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:10:20.381285 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:20.381264    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:10:29.298966 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:29.298920    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:10:39.299520 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:39.299486    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:10:46.561214 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.561179    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"]
May 20 14:10:46.561797 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.561524    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" containerID="cri-o://241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27" gracePeriod=30
May 20 14:10:46.561797 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.561563    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kube-rbac-proxy" containerID="cri-o://56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379" gracePeriod=30
May 20 14:10:46.669647 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.669619    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"]
May 20 14:10:46.670017 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.670004    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kube-rbac-proxy"
May 20 14:10:46.670067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.670019    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kube-rbac-proxy"
May 20 14:10:46.670067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.670033    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="storage-initializer"
May 20 14:10:46.670067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.670039    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="storage-initializer"
May 20 14:10:46.670067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.670056    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container"
May 20 14:10:46.670067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.670062    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container"
May 20 14:10:46.670233 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.670113    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kube-rbac-proxy"
May 20 14:10:46.670233 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.670132    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="d5113ae0-dfae-418e-a047-ddd6b83b3eb0" containerName="kserve-container"
May 20 14:10:46.672342 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.672326    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.674969 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.674943    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
May 20 14:10:46.675090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.675029    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
May 20 14:10:46.683265 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.683240    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"]
May 20 14:10:46.805344 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.805319    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da8d3ed8-31b4-4686-8a85-567bc116dd81-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.805450 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.805347    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.805450 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.805434    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da8d3ed8-31b4-4686-8a85-567bc116dd81-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.805521 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.805462    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/da8d3ed8-31b4-4686-8a85-567bc116dd81-kube-api-access-zs7ms\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.805521 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.805493    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.906352 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.906323    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da8d3ed8-31b4-4686-8a85-567bc116dd81-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.906478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.906367    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/da8d3ed8-31b4-4686-8a85-567bc116dd81-kube-api-access-zs7ms\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.906478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.906411    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.906478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.906454    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da8d3ed8-31b4-4686-8a85-567bc116dd81-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.906478 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:10:46.906458    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
May 20 14:10:46.906676 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.906482    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.906676 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:10:46.906538    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8d3ed8-31b4-4686-8a85-567bc116dd81-proxy-tls podName:da8d3ed8-31b4-4686-8a85-567bc116dd81 nodeName:}" failed. No retries permitted until 2026-05-20 14:10:47.406514545 +0000 UTC m=+1227.544068747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/da8d3ed8-31b4-4686-8a85-567bc116dd81-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" (UID: "da8d3ed8-31b4-4686-8a85-567bc116dd81") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
May 20 14:10:46.906884 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.906846    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da8d3ed8-31b4-4686-8a85-567bc116dd81-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.907061 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.907044    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.907100 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.907054    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:46.915390 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:46.915371    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/da8d3ed8-31b4-4686-8a85-567bc116dd81-kube-api-access-zs7ms\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:47.410919 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:47.410883    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da8d3ed8-31b4-4686-8a85-567bc116dd81-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:47.413237 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:47.413218    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da8d3ed8-31b4-4686-8a85-567bc116dd81-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:47.583186 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:47.583152    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:10:47.597597 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:47.597568    2557 generic.go:358] "Generic (PLEG): container finished" podID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerID="56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379" exitCode=2
May 20 14:10:47.597714 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:47.597647    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" event={"ID":"b0f3fb95-2fd7-487a-b619-2f813d3f79e9","Type":"ContainerDied","Data":"56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379"}
May 20 14:10:47.718094 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:47.718071    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"]
May 20 14:10:47.719940 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:10:47.719909    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8d3ed8_31b4_4686_8a85_567bc116dd81.slice/crio-537089b60e6e1ccf8d2af87bc09799cc84b617166acc037c8a5e9ad08f75b985 WatchSource:0}: Error finding container 537089b60e6e1ccf8d2af87bc09799cc84b617166acc037c8a5e9ad08f75b985: Status 404 returned error can't find the container with id 537089b60e6e1ccf8d2af87bc09799cc84b617166acc037c8a5e9ad08f75b985
May 20 14:10:48.608454 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:48.608413    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" event={"ID":"da8d3ed8-31b4-4686-8a85-567bc116dd81","Type":"ContainerStarted","Data":"7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece"}
May 20 14:10:48.608454 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:48.608457    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" event={"ID":"da8d3ed8-31b4-4686-8a85-567bc116dd81","Type":"ContainerStarted","Data":"537089b60e6e1ccf8d2af87bc09799cc84b617166acc037c8a5e9ad08f75b985"}
May 20 14:10:49.295035 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:49.294995    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
May 20 14:10:49.299831 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:49.299799    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 20 14:10:50.606357 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.606336    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:10:50.616986 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.616960    2557 generic.go:358] "Generic (PLEG): container finished" podID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerID="241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27" exitCode=0
May 20 14:10:50.617100 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.617038    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"
May 20 14:10:50.617100 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.617057    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" event={"ID":"b0f3fb95-2fd7-487a-b619-2f813d3f79e9","Type":"ContainerDied","Data":"241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27"}
May 20 14:10:50.617100 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.617085    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg" event={"ID":"b0f3fb95-2fd7-487a-b619-2f813d3f79e9","Type":"ContainerDied","Data":"8405663156e53f9ba61ca988032f17e77c089ac8ea2a59b43077f81fca4dd9be"}
May 20 14:10:50.617265 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.617101    2557 scope.go:117] "RemoveContainer" containerID="56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379"
May 20 14:10:50.624599 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.624584    2557 scope.go:117] "RemoveContainer" containerID="241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27"
May 20 14:10:50.635134 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.635108    2557 scope.go:117] "RemoveContainer" containerID="7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed"
May 20 14:10:50.643801 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.643775    2557 scope.go:117] "RemoveContainer" containerID="56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379"
May 20 14:10:50.644097 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:10:50.644078    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379\": container with ID starting with 56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379 not found: ID does not exist" containerID="56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379"
May 20 14:10:50.644185 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.644108    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379"} err="failed to get container status \"56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379\": rpc error: code = NotFound desc = could not find container \"56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379\": container with ID starting with 56b2ea757610183be25745d28abb5314bcb6cc3e456fcf1243d0d69cbcacc379 not found: ID does not exist"
May 20 14:10:50.644185 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.644132    2557 scope.go:117] "RemoveContainer" containerID="241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27"
May 20 14:10:50.644371 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:10:50.644355    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27\": container with ID starting with 241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27 not found: ID does not exist" containerID="241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27"
May 20 14:10:50.644436 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.644378    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27"} err="failed to get container status \"241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27\": rpc error: code = NotFound desc = could not find container \"241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27\": container with ID starting with 241af6542b0f0c385f8b30927180260dbc247dac90152ca11a8fcf8694784d27 not found: ID does not exist"
May 20 14:10:50.644436 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.644399    2557 scope.go:117] "RemoveContainer" containerID="7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed"
May 20 14:10:50.644580 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:10:50.644564    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed\": container with ID starting with 7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed not found: ID does not exist" containerID="7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed"
May 20 14:10:50.644633 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.644586    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed"} err="failed to get container status \"7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed\": rpc error: code = NotFound desc = could not find container \"7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed\": container with ID starting with 7f501cdf180daad0945452556355980f540c8665ca0f171e8b07aabf2e8d99ed not found: ID does not exist"
May 20 14:10:50.733790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.733765    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-proxy-tls\") pod \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") "
May 20 14:10:50.733907 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.733839    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqrqp\" (UniqueName: \"kubernetes.io/projected/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kube-api-access-bqrqp\") pod \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") "
May 20 14:10:50.733984 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.733902    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-cabundle-cert\") pod \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") "
May 20 14:10:50.733984 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.733946    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kserve-provision-location\") pod \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") "
May 20 14:10:50.734105 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.733990    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\" (UID: \"b0f3fb95-2fd7-487a-b619-2f813d3f79e9\") "
May 20 14:10:50.734347 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.734316    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0f3fb95-2fd7-487a-b619-2f813d3f79e9" (UID: "b0f3fb95-2fd7-487a-b619-2f813d3f79e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:50.734450 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.734329    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0f3fb95-2fd7-487a-b619-2f813d3f79e9" (UID: "b0f3fb95-2fd7-487a-b619-2f813d3f79e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:10:50.734450 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.734401    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "b0f3fb95-2fd7-487a-b619-2f813d3f79e9" (UID: "b0f3fb95-2fd7-487a-b619-2f813d3f79e9"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:10:50.735762 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.735743    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kube-api-access-bqrqp" (OuterVolumeSpecName: "kube-api-access-bqrqp") pod "b0f3fb95-2fd7-487a-b619-2f813d3f79e9" (UID: "b0f3fb95-2fd7-487a-b619-2f813d3f79e9"). InnerVolumeSpecName "kube-api-access-bqrqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:10:50.735955 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.735936    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0f3fb95-2fd7-487a-b619-2f813d3f79e9" (UID: "b0f3fb95-2fd7-487a-b619-2f813d3f79e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:10:50.835065 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.835031    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:10:50.835065 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.835065    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:10:50.835065 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.835075    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqrqp\" (UniqueName: \"kubernetes.io/projected/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kube-api-access-bqrqp\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:10:50.835218 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.835084    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:10:50.835218 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.835094    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f3fb95-2fd7-487a-b619-2f813d3f79e9-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:10:50.941355 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.941322    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"]
May 20 14:10:50.948296 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:50.948268    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-jjxpg"]
May 20 14:10:51.621625 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:51.621591    2557 generic.go:358] "Generic (PLEG): container finished" podID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerID="7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece" exitCode=0
May 20 14:10:51.622127 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:51.621660    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" event={"ID":"da8d3ed8-31b4-4686-8a85-567bc116dd81","Type":"ContainerDied","Data":"7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece"}
May 20 14:10:52.409902 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:10:52.408639    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" path="/var/lib/kubelet/pods/b0f3fb95-2fd7-487a-b619-2f813d3f79e9/volumes"
May 20 14:11:20.388051 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:11:20.388017    2557 scope.go:117] "RemoveContainer" containerID="f7d5a9e22e07d5942dadf7847876a5c05ccb98540697362685a09b7230e11d9f"
May 20 14:11:20.420669 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:11:20.395953    2557 scope.go:117] "RemoveContainer" containerID="401849cb70a68b29f47b77e26a181ccab1bf3e0737bedff7e6c8153eb7011000"
May 20 14:11:20.420669 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:11:20.404564    2557 scope.go:117] "RemoveContainer" containerID="6495637a36960cfa603378844d69f53c0166442be9d10e0e35c7f5eda6c93bb5"
May 20 14:11:20.420669 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:11:20.412235    2557 scope.go:117] "RemoveContainer" containerID="4fa357526814775d702e79debf010f4e86cd1664d63fea69bc957163b057d7b9"
May 20 14:13:03.346282 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:03.346264    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:13:04.108512 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:04.108478    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" event={"ID":"da8d3ed8-31b4-4686-8a85-567bc116dd81","Type":"ContainerStarted","Data":"fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a"}
May 20 14:13:04.108512 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:04.108513    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" event={"ID":"da8d3ed8-31b4-4686-8a85-567bc116dd81","Type":"ContainerStarted","Data":"bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f"}
May 20 14:13:04.108724 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:04.108638    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:13:05.112744 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:05.112709    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:13:11.121248 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:11.121215    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:13:11.147408 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:11.147354    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" podStartSLOduration=13.542908375 podStartE2EDuration="2m25.147340549s" podCreationTimestamp="2026-05-20 14:10:46 +0000 UTC" firstStartedPulling="2026-05-20 14:10:51.623383548 +0000 UTC m=+1231.760937762" lastFinishedPulling="2026-05-20 14:13:03.227815725 +0000 UTC m=+1363.365369936" observedRunningTime="2026-05-20 14:13:04.152705698 +0000 UTC m=+1364.290259916" watchObservedRunningTime="2026-05-20 14:13:11.147340549 +0000 UTC m=+1371.284894766"
May 20 14:13:20.436375 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:20.436340    2557 scope.go:117] "RemoveContainer" containerID="9f61183a14cfe369ac38591bbca3afc562a08bb98d5cbeebeba3802fda73a4d8"
May 20 14:13:20.444664 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:20.444641    2557 scope.go:117] "RemoveContainer" containerID="10364ea9d8337f589c9faa30944bf7a7f29b07196e285876a58710838721d65a"
May 20 14:13:20.452384 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:20.452356    2557 scope.go:117] "RemoveContainer" containerID="5f98b90c30bf6e6780e03d1282b436f87c68eb5d5ecbcf67a521590e972c6b81"
May 20 14:13:20.459337 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:20.459319    2557 scope.go:117] "RemoveContainer" containerID="755e680cb44e0b6626011ae64a8238aecc6d1350e41aa3598f77c475f5885f15"
May 20 14:13:41.125240 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:41.125208    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:13:46.920390 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:46.920354    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"]
May 20 14:13:46.920814 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:46.920730    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="kserve-container" containerID="cri-o://bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f" gracePeriod=30
May 20 14:13:46.920882 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:46.920805    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="kube-rbac-proxy" containerID="cri-o://fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a" gracePeriod=30
May 20 14:13:47.057696 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.057661    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"]
May 20 14:13:47.058134 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.058110    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kube-rbac-proxy"
May 20 14:13:47.058241 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.058136    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kube-rbac-proxy"
May 20 14:13:47.058241 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.058148    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="storage-initializer"
May 20 14:13:47.058241 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.058156    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="storage-initializer"
May 20 14:13:47.058241 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.058174    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container"
May 20 14:13:47.058241 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.058182    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container"
May 20 14:13:47.058494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.058289    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kserve-container"
May 20 14:13:47.058494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.058308    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0f3fb95-2fd7-487a-b619-2f813d3f79e9" containerName="kube-rbac-proxy"
May 20 14:13:47.061511 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.061481    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.064938 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.064913    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
May 20 14:13:47.065261 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.064929    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
May 20 14:13:47.075388 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.075364    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"]
May 20 14:13:47.110458 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.110428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.110620 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.110468    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.110620 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.110495    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.110620 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.110561    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.110723 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.110670    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qktg9\" (UniqueName: \"kubernetes.io/projected/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kube-api-access-qktg9\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.211403 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.211327    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.211403 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.211368    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.211403 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.211386    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.211663 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.211407    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.211663 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.211468    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qktg9\" (UniqueName: \"kubernetes.io/projected/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kube-api-access-qktg9\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.211975 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.211948    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.212217 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.212196    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.212258 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.212196    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.213778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.213750    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.220759 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.220739    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktg9\" (UniqueName: \"kubernetes.io/projected/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kube-api-access-qktg9\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.251370 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.251343    2557 generic.go:358] "Generic (PLEG): container finished" podID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerID="fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a" exitCode=2
May 20 14:13:47.251456 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.251376    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" event={"ID":"da8d3ed8-31b4-4686-8a85-567bc116dd81","Type":"ContainerDied","Data":"fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a"}
May 20 14:13:47.376658 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.376630    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:47.505761 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.505677    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"]
May 20 14:13:47.508901 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:13:47.508873    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ed9ea1_f58c_4b1a_a8a7_408535b3aa7b.slice/crio-f645502971944a6f6ea161de8314ff5908e1753f20788fff3233970d64a66d84 WatchSource:0}: Error finding container f645502971944a6f6ea161de8314ff5908e1753f20788fff3233970d64a66d84: Status 404 returned error can't find the container with id f645502971944a6f6ea161de8314ff5908e1753f20788fff3233970d64a66d84
May 20 14:13:47.856550 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.856528    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:13:47.918104 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.918079    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/da8d3ed8-31b4-4686-8a85-567bc116dd81-kube-api-access-zs7ms\") pod \"da8d3ed8-31b4-4686-8a85-567bc116dd81\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") "
May 20 14:13:47.918265 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.918123    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da8d3ed8-31b4-4686-8a85-567bc116dd81-proxy-tls\") pod \"da8d3ed8-31b4-4686-8a85-567bc116dd81\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") "
May 20 14:13:47.918265 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.918157    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"da8d3ed8-31b4-4686-8a85-567bc116dd81\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") "
May 20 14:13:47.918265 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.918209    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-cabundle-cert\") pod \"da8d3ed8-31b4-4686-8a85-567bc116dd81\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") "
May 20 14:13:47.918429 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.918316    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da8d3ed8-31b4-4686-8a85-567bc116dd81-kserve-provision-location\") pod \"da8d3ed8-31b4-4686-8a85-567bc116dd81\" (UID: \"da8d3ed8-31b4-4686-8a85-567bc116dd81\") "
May 20 14:13:47.918522 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.918494    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "da8d3ed8-31b4-4686-8a85-567bc116dd81" (UID: "da8d3ed8-31b4-4686-8a85-567bc116dd81"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:13:47.918605 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.918580    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:13:47.918731 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.918700    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8d3ed8-31b4-4686-8a85-567bc116dd81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da8d3ed8-31b4-4686-8a85-567bc116dd81" (UID: "da8d3ed8-31b4-4686-8a85-567bc116dd81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:13:47.918809 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.918711    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da8d3ed8-31b4-4686-8a85-567bc116dd81" (UID: "da8d3ed8-31b4-4686-8a85-567bc116dd81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:13:47.920478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.920459    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8d3ed8-31b4-4686-8a85-567bc116dd81-kube-api-access-zs7ms" (OuterVolumeSpecName: "kube-api-access-zs7ms") pod "da8d3ed8-31b4-4686-8a85-567bc116dd81" (UID: "da8d3ed8-31b4-4686-8a85-567bc116dd81"). InnerVolumeSpecName "kube-api-access-zs7ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:13:47.920930 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:47.920671    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8d3ed8-31b4-4686-8a85-567bc116dd81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da8d3ed8-31b4-4686-8a85-567bc116dd81" (UID: "da8d3ed8-31b4-4686-8a85-567bc116dd81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:13:48.019533 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.019501    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da8d3ed8-31b4-4686-8a85-567bc116dd81-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:13:48.019637 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.019534    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs7ms\" (UniqueName: \"kubernetes.io/projected/da8d3ed8-31b4-4686-8a85-567bc116dd81-kube-api-access-zs7ms\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:13:48.019637 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.019551    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da8d3ed8-31b4-4686-8a85-567bc116dd81-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:13:48.019637 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.019565    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da8d3ed8-31b4-4686-8a85-567bc116dd81-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:13:48.256193 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.256158    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" event={"ID":"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b","Type":"ContainerStarted","Data":"5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e"}
May 20 14:13:48.256335 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.256204    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" event={"ID":"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b","Type":"ContainerStarted","Data":"f645502971944a6f6ea161de8314ff5908e1753f20788fff3233970d64a66d84"}
May 20 14:13:48.257782 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.257752    2557 generic.go:358] "Generic (PLEG): container finished" podID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerID="bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f" exitCode=0
May 20 14:13:48.257900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.257786    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" event={"ID":"da8d3ed8-31b4-4686-8a85-567bc116dd81","Type":"ContainerDied","Data":"bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f"}
May 20 14:13:48.257900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.257829    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5" event={"ID":"da8d3ed8-31b4-4686-8a85-567bc116dd81","Type":"ContainerDied","Data":"537089b60e6e1ccf8d2af87bc09799cc84b617166acc037c8a5e9ad08f75b985"}
May 20 14:13:48.257900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.257839    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"
May 20 14:13:48.257900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.257850    2557 scope.go:117] "RemoveContainer" containerID="fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a"
May 20 14:13:48.266089 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.266072    2557 scope.go:117] "RemoveContainer" containerID="bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f"
May 20 14:13:48.273263 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.273246    2557 scope.go:117] "RemoveContainer" containerID="7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece"
May 20 14:13:48.282416 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.282401    2557 scope.go:117] "RemoveContainer" containerID="fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a"
May 20 14:13:48.282794 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:13:48.282775    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a\": container with ID starting with fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a not found: ID does not exist" containerID="fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a"
May 20 14:13:48.282875 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.282801    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a"} err="failed to get container status \"fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a\": rpc error: code = NotFound desc = could not find container \"fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a\": container with ID starting with fcc244db3db7595d03b2393cb41f027e2b87bc62d281d3130b736943bd09a06a not found: ID does not exist"
May 20 14:13:48.282875 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.282821    2557 scope.go:117] "RemoveContainer" containerID="bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f"
May 20 14:13:48.283195 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:13:48.283177    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f\": container with ID starting with bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f not found: ID does not exist" containerID="bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f"
May 20 14:13:48.283267 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.283203    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f"} err="failed to get container status \"bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f\": rpc error: code = NotFound desc = could not find container \"bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f\": container with ID starting with bd30d59925535b5f85aeff89c43e6fb9494c12ca0f4a8c6278ec3d462070329f not found: ID does not exist"
May 20 14:13:48.283267 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.283224    2557 scope.go:117] "RemoveContainer" containerID="7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece"
May 20 14:13:48.283493 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:13:48.283475    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece\": container with ID starting with 7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece not found: ID does not exist" containerID="7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece"
May 20 14:13:48.283573 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.283498    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece"} err="failed to get container status \"7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece\": rpc error: code = NotFound desc = could not find container \"7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece\": container with ID starting with 7223e91aedc20ac22ab5504ad1a9f6c5a65cf7e0a8a32dbbd68f358a52d29ece not found: ID does not exist"
May 20 14:13:48.288353 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.288330    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"]
May 20 14:13:48.291672 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.291652    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6xxp5"]
May 20 14:13:48.405575 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:48.405542    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" path="/var/lib/kubelet/pods/da8d3ed8-31b4-4686-8a85-567bc116dd81/volumes"
May 20 14:13:52.274698 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:52.274661    2557 generic.go:358] "Generic (PLEG): container finished" podID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerID="5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e" exitCode=0
May 20 14:13:52.274698 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:52.274705    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" event={"ID":"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b","Type":"ContainerDied","Data":"5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e"}
May 20 14:13:53.279815 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:53.279780    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" event={"ID":"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b","Type":"ContainerStarted","Data":"ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551"}
May 20 14:13:53.279815 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:53.279820    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" event={"ID":"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b","Type":"ContainerStarted","Data":"c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289"}
May 20 14:13:53.280239 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:53.280169    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:53.280296 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:53.280277    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:13:53.281295 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:53.281268    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 14:13:53.300342 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:53.300301    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" podStartSLOduration=6.300290385 podStartE2EDuration="6.300290385s" podCreationTimestamp="2026-05-20 14:13:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:13:53.298822843 +0000 UTC m=+1413.436377062" watchObservedRunningTime="2026-05-20 14:13:53.300290385 +0000 UTC m=+1413.437844678"
May 20 14:13:54.284022 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:54.283983    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 14:13:55.287329 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:13:55.287284    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 20 14:14:00.291947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:00.291906    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:14:00.293073 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:00.293051    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:14:07.295601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.295507    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"]
May 20 14:14:07.296008 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.295903    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="kserve-container"
May 20 14:14:07.296008 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.295917    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="kserve-container"
May 20 14:14:07.296008 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.295929    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="kube-rbac-proxy"
May 20 14:14:07.296008 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.295934    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="kube-rbac-proxy"
May 20 14:14:07.296008 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.295942    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="storage-initializer"
May 20 14:14:07.296008 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.295948    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="storage-initializer"
May 20 14:14:07.296222 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.296016    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="kube-rbac-proxy"
May 20 14:14:07.296222 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.296027    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="da8d3ed8-31b4-4686-8a85-567bc116dd81" containerName="kserve-container"
May 20 14:14:07.298420 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.298401    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.303315 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.303290    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
May 20 14:14:07.303460 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.303298    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
May 20 14:14:07.315015 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.314985    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"]
May 20 14:14:07.357278 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.357248    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"]
May 20 14:14:07.357599 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.357546    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kserve-container" containerID="cri-o://c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289" gracePeriod=30
May 20 14:14:07.357664 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.357558    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kube-rbac-proxy" containerID="cri-o://ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551" gracePeriod=30
May 20 14:14:07.365130 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.365105    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.365282 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.365255    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d4bab2e-6914-4009-b9a7-dc853b39d80d-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.365349 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.365300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.365406 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.365388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt626\" (UniqueName: \"kubernetes.io/projected/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kube-api-access-xt626\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.365457 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.365440    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.466433 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.466409    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xt626\" (UniqueName: \"kubernetes.io/projected/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kube-api-access-xt626\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.466528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.466445    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.466528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.466477    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.466628 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.466539    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d4bab2e-6914-4009-b9a7-dc853b39d80d-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.466628 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.466564    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.466973 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.466952    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.467250 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.467229    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.467296 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.467229    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.468793 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.468758    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d4bab2e-6914-4009-b9a7-dc853b39d80d-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.475796 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.475773    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt626\" (UniqueName: \"kubernetes.io/projected/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kube-api-access-xt626\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.609495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.609457    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:07.742690 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.742664    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"]
May 20 14:14:07.745096 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:14:07.745065    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4bab2e_6914_4009_b9a7_dc853b39d80d.slice/crio-c6bc6a7746fe61fc1df96b550ee61517111a72add3998c8fe9ffab16b0295742 WatchSource:0}: Error finding container c6bc6a7746fe61fc1df96b550ee61517111a72add3998c8fe9ffab16b0295742: Status 404 returned error can't find the container with id c6bc6a7746fe61fc1df96b550ee61517111a72add3998c8fe9ffab16b0295742
May 20 14:14:07.996072 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:07.996051    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:14:08.072293 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072267    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kserve-provision-location\") pod \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") "
May 20 14:14:08.072440 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072327    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") "
May 20 14:14:08.072440 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072354    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qktg9\" (UniqueName: \"kubernetes.io/projected/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kube-api-access-qktg9\") pod \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") "
May 20 14:14:08.072440 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072386    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-cabundle-cert\") pod \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") "
May 20 14:14:08.072611 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072474    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-proxy-tls\") pod \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\" (UID: \"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b\") "
May 20 14:14:08.072670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072624    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" (UID: "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:14:08.072670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072632    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" (UID: "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:14:08.072784 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072749    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:08.072832 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072779    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" (UID: "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:14:08.072832 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.072790    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:08.074432 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.074411    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" (UID: "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:14:08.074499 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.074411    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kube-api-access-qktg9" (OuterVolumeSpecName: "kube-api-access-qktg9") pod "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" (UID: "f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b"). InnerVolumeSpecName "kube-api-access-qktg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:14:08.174206 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.174153    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:08.174206 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.174173    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qktg9\" (UniqueName: \"kubernetes.io/projected/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-kube-api-access-qktg9\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:08.174206 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.174183    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:08.334481 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.334455    2557 generic.go:358] "Generic (PLEG): container finished" podID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerID="ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551" exitCode=2
May 20 14:14:08.334481 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.334479    2557 generic.go:358] "Generic (PLEG): container finished" podID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerID="c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289" exitCode=0
May 20 14:14:08.334916 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.334542    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" event={"ID":"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b","Type":"ContainerDied","Data":"ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551"}
May 20 14:14:08.334916 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.334558    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"
May 20 14:14:08.334916 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.334586    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" event={"ID":"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b","Type":"ContainerDied","Data":"c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289"}
May 20 14:14:08.334916 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.334605    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh" event={"ID":"f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b","Type":"ContainerDied","Data":"f645502971944a6f6ea161de8314ff5908e1753f20788fff3233970d64a66d84"}
May 20 14:14:08.334916 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.334624    2557 scope.go:117] "RemoveContainer" containerID="ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551"
May 20 14:14:08.335975 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.335949    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" event={"ID":"5d4bab2e-6914-4009-b9a7-dc853b39d80d","Type":"ContainerStarted","Data":"d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a"}
May 20 14:14:08.336071 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.335982    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" event={"ID":"5d4bab2e-6914-4009-b9a7-dc853b39d80d","Type":"ContainerStarted","Data":"c6bc6a7746fe61fc1df96b550ee61517111a72add3998c8fe9ffab16b0295742"}
May 20 14:14:08.342366 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.342341    2557 scope.go:117] "RemoveContainer" containerID="c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289"
May 20 14:14:08.349715 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.349698    2557 scope.go:117] "RemoveContainer" containerID="5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e"
May 20 14:14:08.359195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.358559    2557 scope.go:117] "RemoveContainer" containerID="ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551"
May 20 14:14:08.359195 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:08.358947    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551\": container with ID starting with ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551 not found: ID does not exist" containerID="ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551"
May 20 14:14:08.359195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.358980    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551"} err="failed to get container status \"ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551\": rpc error: code = NotFound desc = could not find container \"ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551\": container with ID starting with ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551 not found: ID does not exist"
May 20 14:14:08.359195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.359003    2557 scope.go:117] "RemoveContainer" containerID="c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289"
May 20 14:14:08.359410 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:08.359272    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289\": container with ID starting with c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289 not found: ID does not exist" containerID="c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289"
May 20 14:14:08.359410 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.359299    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289"} err="failed to get container status \"c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289\": rpc error: code = NotFound desc = could not find container \"c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289\": container with ID starting with c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289 not found: ID does not exist"
May 20 14:14:08.359410 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.359320    2557 scope.go:117] "RemoveContainer" containerID="5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e"
May 20 14:14:08.359580 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:08.359563    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e\": container with ID starting with 5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e not found: ID does not exist" containerID="5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e"
May 20 14:14:08.359624 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.359597    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e"} err="failed to get container status \"5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e\": rpc error: code = NotFound desc = could not find container \"5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e\": container with ID starting with 5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e not found: ID does not exist"
May 20 14:14:08.359624 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.359616    2557 scope.go:117] "RemoveContainer" containerID="ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551"
May 20 14:14:08.359920 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.359891    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551"} err="failed to get container status \"ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551\": rpc error: code = NotFound desc = could not find container \"ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551\": container with ID starting with ddbd65f2760cbabfa53404e592727c543451f15619fd2a403f70228c83d48551 not found: ID does not exist"
May 20 14:14:08.360001 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.359922    2557 scope.go:117] "RemoveContainer" containerID="c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289"
May 20 14:14:08.360613 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.360595    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289"} err="failed to get container status \"c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289\": rpc error: code = NotFound desc = could not find container \"c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289\": container with ID starting with c370326fce01752513a9381f2d9608fd011052eed54f3d934bd794d60294a289 not found: ID does not exist"
May 20 14:14:08.360677 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.360613    2557 scope.go:117] "RemoveContainer" containerID="5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e"
May 20 14:14:08.360846 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.360830    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e"} err="failed to get container status \"5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e\": rpc error: code = NotFound desc = could not find container \"5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e\": container with ID starting with 5e22625cb8d5eb133363d6026591d7e4902facb9e51a640d21e90d02778ad87e not found: ID does not exist"
May 20 14:14:08.371968 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.371945    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"]
May 20 14:14:08.375501 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.375480    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-g6phh"]
May 20 14:14:08.404554 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:08.404534    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" path="/var/lib/kubelet/pods/f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b/volumes"
May 20 14:14:12.355128 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:12.355086    2557 generic.go:358] "Generic (PLEG): container finished" podID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerID="d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a" exitCode=0
May 20 14:14:12.355695 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:12.355148    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" event={"ID":"5d4bab2e-6914-4009-b9a7-dc853b39d80d","Type":"ContainerDied","Data":"d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a"}
May 20 14:14:13.361899 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:13.361843    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" event={"ID":"5d4bab2e-6914-4009-b9a7-dc853b39d80d","Type":"ContainerStarted","Data":"1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f"}
May 20 14:14:13.362264 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:13.361910    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" event={"ID":"5d4bab2e-6914-4009-b9a7-dc853b39d80d","Type":"ContainerStarted","Data":"f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3"}
May 20 14:14:13.362264 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:13.362135    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:13.393880 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:13.393817    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" podStartSLOduration=6.393801267 podStartE2EDuration="6.393801267s" podCreationTimestamp="2026-05-20 14:14:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:14:13.393771176 +0000 UTC m=+1433.531325393" watchObservedRunningTime="2026-05-20 14:14:13.393801267 +0000 UTC m=+1433.531355486"
May 20 14:14:14.365212 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:14.365172    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:20.373381 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:20.373352    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:50.378016 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:50.377976    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:57.257323 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.257285    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"]
May 20 14:14:57.257958 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.257612    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="kserve-container" containerID="cri-o://f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3" gracePeriod=30
May 20 14:14:57.257958 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.257671    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="kube-rbac-proxy" containerID="cri-o://1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f" gracePeriod=30
May 20 14:14:57.368537 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.368495    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"]
May 20 14:14:57.368972 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.368954    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kserve-container"
May 20 14:14:57.369069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.368974    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kserve-container"
May 20 14:14:57.369069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.368998    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kube-rbac-proxy"
May 20 14:14:57.369069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.369007    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kube-rbac-proxy"
May 20 14:14:57.369069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.369021    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="storage-initializer"
May 20 14:14:57.369069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.369030    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="storage-initializer"
May 20 14:14:57.369339 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.369103    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kserve-container"
May 20 14:14:57.369339 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.369122    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5ed9ea1-f58c-4b1a-a8a7-408535b3aa7b" containerName="kube-rbac-proxy"
May 20 14:14:57.371496 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.371474    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.374251 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.374228    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
May 20 14:14:57.374353 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.374228    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
May 20 14:14:57.383548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.383523    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"]
May 20 14:14:57.456508 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.456470    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.456508 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.456511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.456760 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.456536    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.456760 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.456677    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xsj4\" (UniqueName: \"kubernetes.io/projected/47d38a3d-4fb8-42be-831d-497716384952-kube-api-access-2xsj4\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.456760 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.456754    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d38a3d-4fb8-42be-831d-497716384952-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.505451 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.505413    2557 generic.go:358] "Generic (PLEG): container finished" podID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerID="1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f" exitCode=2
May 20 14:14:57.505637 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.505476    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" event={"ID":"5d4bab2e-6914-4009-b9a7-dc853b39d80d","Type":"ContainerDied","Data":"1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f"}
May 20 14:14:57.557891 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.557765    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2xsj4\" (UniqueName: \"kubernetes.io/projected/47d38a3d-4fb8-42be-831d-497716384952-kube-api-access-2xsj4\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.557891 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.557816    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d38a3d-4fb8-42be-831d-497716384952-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.557891 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.557879    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.558270 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.557912    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.558270 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.557950    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.558270 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:57.558069    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 20 14:14:57.558270 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:57.558139    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls podName:47d38a3d-4fb8-42be-831d-497716384952 nodeName:}" failed. No retries permitted until 2026-05-20 14:14:58.058112209 +0000 UTC m=+1478.195666418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls") pod "isvc-sklearn-mcp-predictor-7445486bb-g7tsz" (UID: "47d38a3d-4fb8-42be-831d-497716384952") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 20 14:14:57.558270 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.558255    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d38a3d-4fb8-42be-831d-497716384952-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.558574 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.558510    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.558574 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.558533    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:57.567060 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:57.567038    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xsj4\" (UniqueName: \"kubernetes.io/projected/47d38a3d-4fb8-42be-831d-497716384952-kube-api-access-2xsj4\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:58.063614 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.063575    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:58.063827 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:58.063733    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 20 14:14:58.063827 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:58.063822    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls podName:47d38a3d-4fb8-42be-831d-497716384952 nodeName:}" failed. No retries permitted until 2026-05-20 14:14:59.063801441 +0000 UTC m=+1479.201355643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls") pod "isvc-sklearn-mcp-predictor-7445486bb-g7tsz" (UID: "47d38a3d-4fb8-42be-831d-497716384952") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 20 14:14:58.318539 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.318517    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:58.366470 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.366440    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") "
May 20 14:14:58.366577 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.366482    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kserve-provision-location\") pod \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") "
May 20 14:14:58.366577 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.366519    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d4bab2e-6914-4009-b9a7-dc853b39d80d-proxy-tls\") pod \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") "
May 20 14:14:58.366577 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.366543    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-cabundle-cert\") pod \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") "
May 20 14:14:58.366577 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.366576    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt626\" (UniqueName: \"kubernetes.io/projected/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kube-api-access-xt626\") pod \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\" (UID: \"5d4bab2e-6914-4009-b9a7-dc853b39d80d\") "
May 20 14:14:58.366974 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.366945    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "5d4bab2e-6914-4009-b9a7-dc853b39d80d" (UID: "5d4bab2e-6914-4009-b9a7-dc853b39d80d"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:14:58.366974 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.366961    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d4bab2e-6914-4009-b9a7-dc853b39d80d" (UID: "5d4bab2e-6914-4009-b9a7-dc853b39d80d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:14:58.367109 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.366976    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d4bab2e-6914-4009-b9a7-dc853b39d80d" (UID: "5d4bab2e-6914-4009-b9a7-dc853b39d80d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:14:58.368526 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.368501    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4bab2e-6914-4009-b9a7-dc853b39d80d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d4bab2e-6914-4009-b9a7-dc853b39d80d" (UID: "5d4bab2e-6914-4009-b9a7-dc853b39d80d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:14:58.368621 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.368602    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kube-api-access-xt626" (OuterVolumeSpecName: "kube-api-access-xt626") pod "5d4bab2e-6914-4009-b9a7-dc853b39d80d" (UID: "5d4bab2e-6914-4009-b9a7-dc853b39d80d"). InnerVolumeSpecName "kube-api-access-xt626". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:14:58.467437 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.467413    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:58.467437 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.467436    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:58.467574 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.467447    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d4bab2e-6914-4009-b9a7-dc853b39d80d-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:58.467574 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.467456    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d4bab2e-6914-4009-b9a7-dc853b39d80d-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:58.467574 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.467464    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt626\" (UniqueName: \"kubernetes.io/projected/5d4bab2e-6914-4009-b9a7-dc853b39d80d-kube-api-access-xt626\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:14:58.510485 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.510462    2557 generic.go:358] "Generic (PLEG): container finished" podID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerID="f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3" exitCode=0
May 20 14:14:58.510572 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.510493    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" event={"ID":"5d4bab2e-6914-4009-b9a7-dc853b39d80d","Type":"ContainerDied","Data":"f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3"}
May 20 14:14:58.510572 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.510513    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk" event={"ID":"5d4bab2e-6914-4009-b9a7-dc853b39d80d","Type":"ContainerDied","Data":"c6bc6a7746fe61fc1df96b550ee61517111a72add3998c8fe9ffab16b0295742"}
May 20 14:14:58.510572 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.510528    2557 scope.go:117] "RemoveContainer" containerID="1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f"
May 20 14:14:58.510572 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.510549    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"
May 20 14:14:58.518529 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.518455    2557 scope.go:117] "RemoveContainer" containerID="f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3"
May 20 14:14:58.525741 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.525724    2557 scope.go:117] "RemoveContainer" containerID="d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a"
May 20 14:14:58.529481 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.529459    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"]
May 20 14:14:58.533280 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.533254    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-8bflk"]
May 20 14:14:58.533679 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.533665    2557 scope.go:117] "RemoveContainer" containerID="1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f"
May 20 14:14:58.533934 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:58.533916    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f\": container with ID starting with 1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f not found: ID does not exist" containerID="1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f"
May 20 14:14:58.533992 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.533941    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f"} err="failed to get container status \"1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f\": rpc error: code = NotFound desc = could not find container \"1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f\": container with ID starting with 1b4d0f4bfe76ac4bf509b4758e11a842bc4961abd6199ec9400b06f6d4a9fb5f not found: ID does not exist"
May 20 14:14:58.533992 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.533958    2557 scope.go:117] "RemoveContainer" containerID="f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3"
May 20 14:14:58.534210 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:58.534191    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3\": container with ID starting with f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3 not found: ID does not exist" containerID="f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3"
May 20 14:14:58.534264 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.534215    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3"} err="failed to get container status \"f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3\": rpc error: code = NotFound desc = could not find container \"f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3\": container with ID starting with f794546a99a0c5c393b0b2f4bdc8cd4a27580d0d6cc5f3c440d378369acdf1a3 not found: ID does not exist"
May 20 14:14:58.534264 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.534236    2557 scope.go:117] "RemoveContainer" containerID="d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a"
May 20 14:14:58.534473 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:14:58.534456    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a\": container with ID starting with d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a not found: ID does not exist" containerID="d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a"
May 20 14:14:58.534524 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:58.534478    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a"} err="failed to get container status \"d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a\": rpc error: code = NotFound desc = could not find container \"d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a\": container with ID starting with d088acc7507399b67ddec98aedbfadabb291a59f383a8d5c183dda631b307b5a not found: ID does not exist"
May 20 14:14:59.072547 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:59.072524    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:59.074929 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:59.074906    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7445486bb-g7tsz\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:59.181916 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:59.181896    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:14:59.312273 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:59.312228    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"]
May 20 14:14:59.314218 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:14:59.314190    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d38a3d_4fb8_42be_831d_497716384952.slice/crio-47654769dc6765cb08e893fca1183470a08149e8d24da3f353a882ee6e13eb7b WatchSource:0}: Error finding container 47654769dc6765cb08e893fca1183470a08149e8d24da3f353a882ee6e13eb7b: Status 404 returned error can't find the container with id 47654769dc6765cb08e893fca1183470a08149e8d24da3f353a882ee6e13eb7b
May 20 14:14:59.516233 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:59.516197    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerStarted","Data":"3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33"}
May 20 14:14:59.516233 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:14:59.516233    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerStarted","Data":"47654769dc6765cb08e893fca1183470a08149e8d24da3f353a882ee6e13eb7b"}
May 20 14:15:00.405932 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:00.405889    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" path="/var/lib/kubelet/pods/5d4bab2e-6914-4009-b9a7-dc853b39d80d/volumes"
May 20 14:15:03.530850 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:03.530811    2557 generic.go:358] "Generic (PLEG): container finished" podID="47d38a3d-4fb8-42be-831d-497716384952" containerID="3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33" exitCode=0
May 20 14:15:03.531318 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:03.530886    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerDied","Data":"3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33"}
May 20 14:15:04.535974 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:04.535942    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerStarted","Data":"b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be"}
May 20 14:15:07.548598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:07.548561    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerStarted","Data":"7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32"}
May 20 14:15:07.548598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:07.548599    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerStarted","Data":"d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131"}
May 20 14:15:07.549074 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:07.548774    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:15:07.572640 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:07.572596    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podStartSLOduration=7.517837952 podStartE2EDuration="10.572581233s" podCreationTimestamp="2026-05-20 14:14:57 +0000 UTC" firstStartedPulling="2026-05-20 14:15:03.604703077 +0000 UTC m=+1483.742257274" lastFinishedPulling="2026-05-20 14:15:06.659446356 +0000 UTC m=+1486.797000555" observedRunningTime="2026-05-20 14:15:07.571068385 +0000 UTC m=+1487.708622602" watchObservedRunningTime="2026-05-20 14:15:07.572581233 +0000 UTC m=+1487.710135450"
May 20 14:15:08.551905 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:08.551874    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:15:08.551905 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:08.551908    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:15:14.561296 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:14.561265    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:15:20.404160 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:20.404133    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:15:20.406537 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:20.406517    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:15:20.409409 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:20.409377    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:15:20.410666 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:20.410647    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:15:34.565985 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:15:34.565882    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:16:14.567364 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:14.567328    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:16:17.449399 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.449365    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"]
May 20 14:16:17.449843 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.449809    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-container" containerID="cri-o://b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be" gracePeriod=30
May 20 14:16:17.449955 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.449869    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-agent" containerID="cri-o://d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131" gracePeriod=30
May 20 14:16:17.450014 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.449870    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy" containerID="cri-o://7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32" gracePeriod=30
May 20 14:16:17.539026 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.539004    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"]
May 20 14:16:17.539370 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.539359    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="kserve-container"
May 20 14:16:17.539419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.539372    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="kserve-container"
May 20 14:16:17.539419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.539386    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="storage-initializer"
May 20 14:16:17.539419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.539391    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="storage-initializer"
May 20 14:16:17.539419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.539401    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="kube-rbac-proxy"
May 20 14:16:17.539419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.539406    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="kube-rbac-proxy"
May 20 14:16:17.539567 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.539457    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="kube-rbac-proxy"
May 20 14:16:17.539567 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.539464    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="5d4bab2e-6914-4009-b9a7-dc853b39d80d" containerName="kserve-container"
May 20 14:16:17.542564 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.542549    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.545321 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.545301    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
May 20 14:16:17.545419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.545366    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
May 20 14:16:17.552662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.552635    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"]
May 20 14:16:17.553543 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.553523    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.553622 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.553555    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq86t\" (UniqueName: \"kubernetes.io/projected/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kube-api-access-wq86t\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.553622 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.553583    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.553697 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.553654    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.553734 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.553700    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-cabundle-cert\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.654498 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.654473    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wq86t\" (UniqueName: \"kubernetes.io/projected/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kube-api-access-wq86t\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.654598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.654503    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.654598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.654525    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.654598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.654550    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-cabundle-cert\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.654747 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.654614    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.654747 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:16:17.654727    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
May 20 14:16:17.654838 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:16:17.654774    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-proxy-tls podName:ae6c170a-8b7a-4437-acc5-7c9618dc3b7a nodeName:}" failed. No retries permitted until 2026-05-20 14:16:18.154755282 +0000 UTC m=+1558.292309478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-proxy-tls") pod "isvc-paddle-predictor-6b8b7cfb4b-7jggq" (UID: "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a") : secret "isvc-paddle-predictor-serving-cert" not found
May 20 14:16:17.654911 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.654893    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.655280 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.655259    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.655319 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.655259    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-cabundle-cert\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.664305 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.664279    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq86t\" (UniqueName: \"kubernetes.io/projected/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kube-api-access-wq86t\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:17.775635 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.775578    2557 generic.go:358] "Generic (PLEG): container finished" podID="47d38a3d-4fb8-42be-831d-497716384952" containerID="7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32" exitCode=2
May 20 14:16:17.775732 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:17.775655    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerDied","Data":"7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32"}
May 20 14:16:18.157911 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:18.157878    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:18.160250 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:18.160220    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-7jggq\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:18.453379 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:18.453292    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:18.589280 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:18.589249    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"]
May 20 14:16:18.593827 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:16:18.593798    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6c170a_8b7a_4437_acc5_7c9618dc3b7a.slice/crio-c6e084e3518dc14ca51be1a7edcc1c4166d6aca91fc703ebfb2845732d90f923 WatchSource:0}: Error finding container c6e084e3518dc14ca51be1a7edcc1c4166d6aca91fc703ebfb2845732d90f923: Status 404 returned error can't find the container with id c6e084e3518dc14ca51be1a7edcc1c4166d6aca91fc703ebfb2845732d90f923
May 20 14:16:18.780968 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:18.780892    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" event={"ID":"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a","Type":"ContainerStarted","Data":"5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611"}
May 20 14:16:18.780968 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:18.780929    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" event={"ID":"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a","Type":"ContainerStarted","Data":"c6e084e3518dc14ca51be1a7edcc1c4166d6aca91fc703ebfb2845732d90f923"}
May 20 14:16:19.555142 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:19.555097    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 14:16:19.786684 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:19.786653    2557 generic.go:358] "Generic (PLEG): container finished" podID="47d38a3d-4fb8-42be-831d-497716384952" containerID="b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be" exitCode=0
May 20 14:16:19.786850 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:19.786709    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerDied","Data":"b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be"}
May 20 14:16:23.802689 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:23.802651    2557 generic.go:358] "Generic (PLEG): container finished" podID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerID="5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611" exitCode=0
May 20 14:16:23.803139 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:23.802738    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" event={"ID":"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a","Type":"ContainerDied","Data":"5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611"}
May 20 14:16:24.555217 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:24.555179    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 14:16:24.563918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:24.563882    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.34:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 14:16:29.555168 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:29.555116    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 14:16:29.555641 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:29.555286    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:16:34.555594 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:34.555474    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 14:16:34.564276 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:34.564245    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.34:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 14:16:34.842468 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:34.842445    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" event={"ID":"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a","Type":"ContainerStarted","Data":"19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8"}
May 20 14:16:35.847347 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:35.847308    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" event={"ID":"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a","Type":"ContainerStarted","Data":"76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05"}
May 20 14:16:35.847783 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:35.847485    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:35.870953 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:35.870907    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" podStartSLOduration=7.900644253 podStartE2EDuration="18.870893795s" podCreationTimestamp="2026-05-20 14:16:17 +0000 UTC" firstStartedPulling="2026-05-20 14:16:23.80416725 +0000 UTC m=+1563.941721450" lastFinishedPulling="2026-05-20 14:16:34.774416777 +0000 UTC m=+1574.911970992" observedRunningTime="2026-05-20 14:16:35.867967028 +0000 UTC m=+1576.005521246" watchObservedRunningTime="2026-05-20 14:16:35.870893795 +0000 UTC m=+1576.008448013"
May 20 14:16:36.850282 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:36.850252    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:36.851374 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:36.851343    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:16:37.853678 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:37.853633    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:16:39.554471 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:39.554439    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 14:16:42.858243 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:42.858212    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:16:42.859136 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:42.859106    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:16:44.554430 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:44.554395    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 20 14:16:44.563844 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:44.563820    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.34:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.34:8080: connect: connection refused"
May 20 14:16:44.563941 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:44.563927    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:16:47.589701 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.589679    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:16:47.690333 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.690312    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls\") pod \"47d38a3d-4fb8-42be-831d-497716384952\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") "
May 20 14:16:47.690463 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.690353    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xsj4\" (UniqueName: \"kubernetes.io/projected/47d38a3d-4fb8-42be-831d-497716384952-kube-api-access-2xsj4\") pod \"47d38a3d-4fb8-42be-831d-497716384952\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") "
May 20 14:16:47.690463 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.690378    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d38a3d-4fb8-42be-831d-497716384952-kserve-provision-location\") pod \"47d38a3d-4fb8-42be-831d-497716384952\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") "
May 20 14:16:47.690463 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.690399    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-cabundle-cert\") pod \"47d38a3d-4fb8-42be-831d-497716384952\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") "
May 20 14:16:47.690463 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.690448    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"47d38a3d-4fb8-42be-831d-497716384952\" (UID: \"47d38a3d-4fb8-42be-831d-497716384952\") "
May 20 14:16:47.690731 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.690706    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d38a3d-4fb8-42be-831d-497716384952-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47d38a3d-4fb8-42be-831d-497716384952" (UID: "47d38a3d-4fb8-42be-831d-497716384952"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:16:47.690836 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.690807    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47d38a3d-4fb8-42be-831d-497716384952" (UID: "47d38a3d-4fb8-42be-831d-497716384952"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:16:47.690925 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.690902    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "47d38a3d-4fb8-42be-831d-497716384952" (UID: "47d38a3d-4fb8-42be-831d-497716384952"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:16:47.692433 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.692409    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47d38a3d-4fb8-42be-831d-497716384952" (UID: "47d38a3d-4fb8-42be-831d-497716384952"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:16:47.692515 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.692442    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d38a3d-4fb8-42be-831d-497716384952-kube-api-access-2xsj4" (OuterVolumeSpecName: "kube-api-access-2xsj4") pod "47d38a3d-4fb8-42be-831d-497716384952" (UID: "47d38a3d-4fb8-42be-831d-497716384952"). InnerVolumeSpecName "kube-api-access-2xsj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:16:47.791279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.791226    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d38a3d-4fb8-42be-831d-497716384952-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:16:47.791279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.791247    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xsj4\" (UniqueName: \"kubernetes.io/projected/47d38a3d-4fb8-42be-831d-497716384952-kube-api-access-2xsj4\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:16:47.791279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.791257    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d38a3d-4fb8-42be-831d-497716384952-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:16:47.791279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.791266    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:16:47.791279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.791275    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d38a3d-4fb8-42be-831d-497716384952-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:16:47.885906 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.885884    2557 generic.go:358] "Generic (PLEG): container finished" podID="47d38a3d-4fb8-42be-831d-497716384952" containerID="d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131" exitCode=0
May 20 14:16:47.886001 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.885935    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerDied","Data":"d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131"}
May 20 14:16:47.886001 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.885955    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz" event={"ID":"47d38a3d-4fb8-42be-831d-497716384952","Type":"ContainerDied","Data":"47654769dc6765cb08e893fca1183470a08149e8d24da3f353a882ee6e13eb7b"}
May 20 14:16:47.886001 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.885971    2557 scope.go:117] "RemoveContainer" containerID="7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32"
May 20 14:16:47.886001 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.885973    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"
May 20 14:16:47.898929 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.895468    2557 scope.go:117] "RemoveContainer" containerID="d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131"
May 20 14:16:47.903361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.903338    2557 scope.go:117] "RemoveContainer" containerID="b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be"
May 20 14:16:47.910466 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.910449    2557 scope.go:117] "RemoveContainer" containerID="3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33"
May 20 14:16:47.917268 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.917244    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"]
May 20 14:16:47.920490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.920471    2557 scope.go:117] "RemoveContainer" containerID="7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32"
May 20 14:16:47.920719 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:16:47.920702    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32\": container with ID starting with 7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32 not found: ID does not exist" containerID="7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32"
May 20 14:16:47.920784 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.920731    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32"} err="failed to get container status \"7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32\": rpc error: code = NotFound desc = could not find container \"7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32\": container with ID starting with 7412afdf5e3f44688eab69501562d409e31c3b08dcaa68f4f768ad95a0d1bc32 not found: ID does not exist"
May 20 14:16:47.920784 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.920749    2557 scope.go:117] "RemoveContainer" containerID="d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131"
May 20 14:16:47.921048 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:16:47.921034    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131\": container with ID starting with d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131 not found: ID does not exist" containerID="d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131"
May 20 14:16:47.921103 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.921050    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131"} err="failed to get container status \"d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131\": rpc error: code = NotFound desc = could not find container \"d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131\": container with ID starting with d4533c0d9514c67fbb1feea33b789c496ff73fba9dc3b9da0168016da11c8131 not found: ID does not exist"
May 20 14:16:47.921103 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.921062    2557 scope.go:117] "RemoveContainer" containerID="b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be"
May 20 14:16:47.921258 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:16:47.921239    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be\": container with ID starting with b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be not found: ID does not exist" containerID="b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be"
May 20 14:16:47.921341 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.921268    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be"} err="failed to get container status \"b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be\": rpc error: code = NotFound desc = could not find container \"b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be\": container with ID starting with b764f4352a9520ef0cfc6de5897e1b43096a4cebf78495bbf848d12cfb8644be not found: ID does not exist"
May 20 14:16:47.921341 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.921280    2557 scope.go:117] "RemoveContainer" containerID="3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33"
May 20 14:16:47.921464 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.921445    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7445486bb-g7tsz"]
May 20 14:16:47.921518 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:16:47.921487    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33\": container with ID starting with 3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33 not found: ID does not exist" containerID="3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33"
May 20 14:16:47.921518 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:47.921503    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33"} err="failed to get container status \"3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33\": rpc error: code = NotFound desc = could not find container \"3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33\": container with ID starting with 3a9051d3d45ad5bdde0649963d9dd855874a128e10e582b11ca2dbc02d0dff33 not found: ID does not exist"
May 20 14:16:48.405239 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:48.405200    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47d38a3d-4fb8-42be-831d-497716384952" path="/var/lib/kubelet/pods/47d38a3d-4fb8-42be-831d-497716384952/volumes"
May 20 14:16:52.859207 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:16:52.859173    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:17:02.859752 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:02.859675    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:17:12.859319 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:12.859278    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 20 14:17:22.860499 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:22.860460    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:17:28.998263 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:28.998228    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"]
May 20 14:17:28.998633 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:28.998536    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container" containerID="cri-o://19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8" gracePeriod=30
May 20 14:17:28.998633 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:28.998582    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kube-rbac-proxy" containerID="cri-o://76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05" gracePeriod=30
May 20 14:17:29.125967 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.125937    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"]
May 20 14:17:29.126359 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126346    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy"
May 20 14:17:29.126403 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126361    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy"
May 20 14:17:29.126403 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126376    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-agent"
May 20 14:17:29.126403 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126381    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-agent"
May 20 14:17:29.126497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126415    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-container"
May 20 14:17:29.126497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126421    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-container"
May 20 14:17:29.126497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126428    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="storage-initializer"
May 20 14:17:29.126497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126433    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="storage-initializer"
May 20 14:17:29.126497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126494    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kube-rbac-proxy"
May 20 14:17:29.126640 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126503    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-container"
May 20 14:17:29.126640 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.126510    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="47d38a3d-4fb8-42be-831d-497716384952" containerName="kserve-agent"
May 20 14:17:29.129661 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.129643    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.133211 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.133193    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
May 20 14:17:29.134058 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.134043    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
May 20 14:17:29.141382 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.141362    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"]
May 20 14:17:29.192439 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.192418    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.192532 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.192452    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.192532 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.192512    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577fz\" (UniqueName: \"kubernetes.io/projected/a66442e9-9658-48c7-abb9-1c7f51fa7448-kube-api-access-577fz\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.192645 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.192548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66442e9-9658-48c7-abb9-1c7f51fa7448-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.192645 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.192583    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a66442e9-9658-48c7-abb9-1c7f51fa7448-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.293134 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.293062    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-577fz\" (UniqueName: \"kubernetes.io/projected/a66442e9-9658-48c7-abb9-1c7f51fa7448-kube-api-access-577fz\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.293134 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.293115    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66442e9-9658-48c7-abb9-1c7f51fa7448-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.293291 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.293154    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a66442e9-9658-48c7-abb9-1c7f51fa7448-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.293291 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.293186    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.293291 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.293213    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.293449 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:17:29.293318    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
May 20 14:17:29.293449 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:17:29.293414    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a66442e9-9658-48c7-abb9-1c7f51fa7448-proxy-tls podName:a66442e9-9658-48c7-abb9-1c7f51fa7448 nodeName:}" failed. No retries permitted until 2026-05-20 14:17:29.793393473 +0000 UTC m=+1629.930947680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a66442e9-9658-48c7-abb9-1c7f51fa7448-proxy-tls") pod "isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" (UID: "a66442e9-9658-48c7-abb9-1c7f51fa7448") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
May 20 14:17:29.293632 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.293614    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a66442e9-9658-48c7-abb9-1c7f51fa7448-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.294010 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.293990    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.294010 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.294000    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.304209 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.304186    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-577fz\" (UniqueName: \"kubernetes.io/projected/a66442e9-9658-48c7-abb9-1c7f51fa7448-kube-api-access-577fz\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.797163 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.797124    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66442e9-9658-48c7-abb9-1c7f51fa7448-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:29.799391 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:29.799364    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66442e9-9658-48c7-abb9-1c7f51fa7448-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:30.023122 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:30.023090    2557 generic.go:358] "Generic (PLEG): container finished" podID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerID="76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05" exitCode=2
May 20 14:17:30.023465 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:30.023167    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" event={"ID":"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a","Type":"ContainerDied","Data":"76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05"}
May 20 14:17:30.039304 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:30.039281    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:30.169663 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:30.169639    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"]
May 20 14:17:30.171652 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:17:30.171624    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66442e9_9658_48c7_abb9_1c7f51fa7448.slice/crio-764a0d3defd02ead3d989cc62c9081195b20a1a5ba7d84c491e18005f542d7bc WatchSource:0}: Error finding container 764a0d3defd02ead3d989cc62c9081195b20a1a5ba7d84c491e18005f542d7bc: Status 404 returned error can't find the container with id 764a0d3defd02ead3d989cc62c9081195b20a1a5ba7d84c491e18005f542d7bc
May 20 14:17:31.027944 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.027908    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" event={"ID":"a66442e9-9658-48c7-abb9-1c7f51fa7448","Type":"ContainerStarted","Data":"95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68"}
May 20 14:17:31.027944 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.027943    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" event={"ID":"a66442e9-9658-48c7-abb9-1c7f51fa7448","Type":"ContainerStarted","Data":"764a0d3defd02ead3d989cc62c9081195b20a1a5ba7d84c491e18005f542d7bc"}
May 20 14:17:31.440274 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.440254    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:17:31.515135 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.515107    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-proxy-tls\") pod \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") "
May 20 14:17:31.515262 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.515142    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-cabundle-cert\") pod \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") "
May 20 14:17:31.515262 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.515185    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kserve-provision-location\") pod \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") "
May 20 14:17:31.515262 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.515238    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq86t\" (UniqueName: \"kubernetes.io/projected/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kube-api-access-wq86t\") pod \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") "
May 20 14:17:31.515427 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.515373    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\" (UID: \"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a\") "
May 20 14:17:31.515638 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.515536    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" (UID: "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:17:31.515733 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.515677    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" (UID: "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:17:31.517163 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.517133    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kube-api-access-wq86t" (OuterVolumeSpecName: "kube-api-access-wq86t") pod "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" (UID: "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a"). InnerVolumeSpecName "kube-api-access-wq86t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:17:31.517419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.517405    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" (UID: "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:17:31.525559 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.525537    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" (UID: "ae6c170a-8b7a-4437-acc5-7c9618dc3b7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:17:31.616244 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.616224    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:17:31.616339 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.616247    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:17:31.616339 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.616260    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:17:31.616339 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.616268    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq86t\" (UniqueName: \"kubernetes.io/projected/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-kube-api-access-wq86t\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:17:31.616339 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:31.616280    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:17:32.033467 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.033396    2557 generic.go:358] "Generic (PLEG): container finished" podID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerID="19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8" exitCode=0
May 20 14:17:32.033790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.033477    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" event={"ID":"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a","Type":"ContainerDied","Data":"19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8"}
May 20 14:17:32.033790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.033504    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"
May 20 14:17:32.033790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.033518    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq" event={"ID":"ae6c170a-8b7a-4437-acc5-7c9618dc3b7a","Type":"ContainerDied","Data":"c6e084e3518dc14ca51be1a7edcc1c4166d6aca91fc703ebfb2845732d90f923"}
May 20 14:17:32.033790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.033538    2557 scope.go:117] "RemoveContainer" containerID="76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05"
May 20 14:17:32.043697 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.043678    2557 scope.go:117] "RemoveContainer" containerID="19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8"
May 20 14:17:32.051580 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.051561    2557 scope.go:117] "RemoveContainer" containerID="5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611"
May 20 14:17:32.058760 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.058683    2557 scope.go:117] "RemoveContainer" containerID="76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05"
May 20 14:17:32.059013 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:17:32.058993    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05\": container with ID starting with 76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05 not found: ID does not exist" containerID="76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05"
May 20 14:17:32.059106 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.059027    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05"} err="failed to get container status \"76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05\": rpc error: code = NotFound desc = could not find container \"76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05\": container with ID starting with 76ff3edc3f4f5034da005a09ca7641e67bccddaadfdf5626a66bb2e328cfbf05 not found: ID does not exist"
May 20 14:17:32.059106 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.059052    2557 scope.go:117] "RemoveContainer" containerID="19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8"
May 20 14:17:32.059381 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:17:32.059364    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8\": container with ID starting with 19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8 not found: ID does not exist" containerID="19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8"
May 20 14:17:32.059464 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.059389    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8"} err="failed to get container status \"19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8\": rpc error: code = NotFound desc = could not find container \"19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8\": container with ID starting with 19756a9aa0d9faa4174ef66e9a110f2e44adc87f0bb0bdebc784237e014dfef8 not found: ID does not exist"
May 20 14:17:32.059464 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.059411    2557 scope.go:117] "RemoveContainer" containerID="5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611"
May 20 14:17:32.059718 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:17:32.059692    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611\": container with ID starting with 5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611 not found: ID does not exist" containerID="5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611"
May 20 14:17:32.059766 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.059723    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611"} err="failed to get container status \"5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611\": rpc error: code = NotFound desc = could not find container \"5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611\": container with ID starting with 5cfd906d04adff6174d092a2d08462cee916fb6a9e3acb93bf8a7206c5921611 not found: ID does not exist"
May 20 14:17:32.072072 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.072049    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"]
May 20 14:17:32.077503 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.077480    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-7jggq"]
May 20 14:17:32.405005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:32.404974    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" path="/var/lib/kubelet/pods/ae6c170a-8b7a-4437-acc5-7c9618dc3b7a/volumes"
May 20 14:17:35.045989 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:35.045949    2557 generic.go:358] "Generic (PLEG): container finished" podID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerID="95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68" exitCode=0
May 20 14:17:35.046362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:35.046033    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" event={"ID":"a66442e9-9658-48c7-abb9-1c7f51fa7448","Type":"ContainerDied","Data":"95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68"}
May 20 14:17:36.054221 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:36.054188    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" event={"ID":"a66442e9-9658-48c7-abb9-1c7f51fa7448","Type":"ContainerStarted","Data":"0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636"}
May 20 14:17:36.054672 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:36.054231    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" event={"ID":"a66442e9-9658-48c7-abb9-1c7f51fa7448","Type":"ContainerStarted","Data":"418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72"}
May 20 14:17:36.054672 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:36.054492    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:36.077713 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:36.077655    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" podStartSLOduration=7.077639204 podStartE2EDuration="7.077639204s" podCreationTimestamp="2026-05-20 14:17:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:17:36.075375536 +0000 UTC m=+1636.212929753" watchObservedRunningTime="2026-05-20 14:17:36.077639204 +0000 UTC m=+1636.215193421"
May 20 14:17:37.057925 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:37.057890    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:37.058874 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:37.058837    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 14:17:38.060528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:38.060489    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 14:17:43.064593 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:43.064564    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:17:43.065238 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:43.065205    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 14:17:53.065823 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:17:53.065788    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 14:18:03.065279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:03.065239    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 14:18:13.065975 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:13.065935    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 20 14:18:23.065777 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:23.065739    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:18:30.606762 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.606727    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"]
May 20 14:18:30.607244 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.607151    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container" containerID="cri-o://418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72" gracePeriod=30
May 20 14:18:30.607329 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.607230    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kube-rbac-proxy" containerID="cri-o://0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636" gracePeriod=30
May 20 14:18:30.732517 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.732487    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"]
May 20 14:18:30.732826 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.732815    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container"
May 20 14:18:30.732900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.732828    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container"
May 20 14:18:30.732900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.732836    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kube-rbac-proxy"
May 20 14:18:30.732900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.732841    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kube-rbac-proxy"
May 20 14:18:30.732900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.732866    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="storage-initializer"
May 20 14:18:30.732900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.732874    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="storage-initializer"
May 20 14:18:30.733073 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.732948    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kserve-container"
May 20 14:18:30.733073 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.732955    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="ae6c170a-8b7a-4437-acc5-7c9618dc3b7a" containerName="kube-rbac-proxy"
May 20 14:18:30.736097 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.736082    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.738841 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.738819    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
May 20 14:18:30.739102 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.739082    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
May 20 14:18:30.749072 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.749052    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"]
May 20 14:18:30.864283 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.864227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33cc1b8e-0933-45e9-9999-1a1881d12d65-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.864373 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.864293    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.864373 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.864321    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.864373 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.864353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cc1b8e-0933-45e9-9999-1a1881d12d65-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.864479 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.864398    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzsvc\" (UniqueName: \"kubernetes.io/projected/33cc1b8e-0933-45e9-9999-1a1881d12d65-kube-api-access-qzsvc\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.964974 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.964949    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.965067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.964992    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cc1b8e-0933-45e9-9999-1a1881d12d65-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.965067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.965009    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qzsvc\" (UniqueName: \"kubernetes.io/projected/33cc1b8e-0933-45e9-9999-1a1881d12d65-kube-api-access-qzsvc\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.965067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.965042    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33cc1b8e-0933-45e9-9999-1a1881d12d65-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.965189 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.965123    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.965519 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.965496    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33cc1b8e-0933-45e9-9999-1a1881d12d65-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.965609 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.965590    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.965717 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.965701    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.967319 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.967298    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cc1b8e-0933-45e9-9999-1a1881d12d65-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:30.974719 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:30.974696    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzsvc\" (UniqueName: \"kubernetes.io/projected/33cc1b8e-0933-45e9-9999-1a1881d12d65-kube-api-access-qzsvc\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:31.045690 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:31.045665    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:31.170522 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:31.170387    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"]
May 20 14:18:31.172778 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:18:31.172753    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cc1b8e_0933_45e9_9999_1a1881d12d65.slice/crio-713c59d88da8b55487d21bd656b6edf0346646c40ac9d6295355c05fbb02c823 WatchSource:0}: Error finding container 713c59d88da8b55487d21bd656b6edf0346646c40ac9d6295355c05fbb02c823: Status 404 returned error can't find the container with id 713c59d88da8b55487d21bd656b6edf0346646c40ac9d6295355c05fbb02c823
May 20 14:18:31.174628 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:31.174614    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:18:31.229707 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:31.229687    2557 generic.go:358] "Generic (PLEG): container finished" podID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerID="0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636" exitCode=2
May 20 14:18:31.229797 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:31.229768    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" event={"ID":"a66442e9-9658-48c7-abb9-1c7f51fa7448","Type":"ContainerDied","Data":"0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636"}
May 20 14:18:31.231112 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:31.231090    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" event={"ID":"33cc1b8e-0933-45e9-9999-1a1881d12d65","Type":"ContainerStarted","Data":"d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79"}
May 20 14:18:31.231205 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:31.231121    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" event={"ID":"33cc1b8e-0933-45e9-9999-1a1881d12d65","Type":"ContainerStarted","Data":"713c59d88da8b55487d21bd656b6edf0346646c40ac9d6295355c05fbb02c823"}
May 20 14:18:32.953428 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:32.953408    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:18:33.081923 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.081898    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a66442e9-9658-48c7-abb9-1c7f51fa7448-kserve-provision-location\") pod \"a66442e9-9658-48c7-abb9-1c7f51fa7448\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") "
May 20 14:18:33.082056 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.081946    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-cabundle-cert\") pod \"a66442e9-9658-48c7-abb9-1c7f51fa7448\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") "
May 20 14:18:33.082056 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.081971    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-577fz\" (UniqueName: \"kubernetes.io/projected/a66442e9-9658-48c7-abb9-1c7f51fa7448-kube-api-access-577fz\") pod \"a66442e9-9658-48c7-abb9-1c7f51fa7448\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") "
May 20 14:18:33.082176 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.082059    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"a66442e9-9658-48c7-abb9-1c7f51fa7448\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") "
May 20 14:18:33.082176 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.082106    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66442e9-9658-48c7-abb9-1c7f51fa7448-proxy-tls\") pod \"a66442e9-9658-48c7-abb9-1c7f51fa7448\" (UID: \"a66442e9-9658-48c7-abb9-1c7f51fa7448\") "
May 20 14:18:33.082411 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.082383    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a66442e9-9658-48c7-abb9-1c7f51fa7448" (UID: "a66442e9-9658-48c7-abb9-1c7f51fa7448"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:18:33.082463 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.082411    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "a66442e9-9658-48c7-abb9-1c7f51fa7448" (UID: "a66442e9-9658-48c7-abb9-1c7f51fa7448"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:18:33.084169 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.084148    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66442e9-9658-48c7-abb9-1c7f51fa7448-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a66442e9-9658-48c7-abb9-1c7f51fa7448" (UID: "a66442e9-9658-48c7-abb9-1c7f51fa7448"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:18:33.084297 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.084278    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66442e9-9658-48c7-abb9-1c7f51fa7448-kube-api-access-577fz" (OuterVolumeSpecName: "kube-api-access-577fz") pod "a66442e9-9658-48c7-abb9-1c7f51fa7448" (UID: "a66442e9-9658-48c7-abb9-1c7f51fa7448"). InnerVolumeSpecName "kube-api-access-577fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:18:33.097987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.097960    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66442e9-9658-48c7-abb9-1c7f51fa7448-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a66442e9-9658-48c7-abb9-1c7f51fa7448" (UID: "a66442e9-9658-48c7-abb9-1c7f51fa7448"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:33.183243 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.183217    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:18:33.183243 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.183241    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a66442e9-9658-48c7-abb9-1c7f51fa7448-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:18:33.183379 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.183255    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a66442e9-9658-48c7-abb9-1c7f51fa7448-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:18:33.183379 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.183269    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a66442e9-9658-48c7-abb9-1c7f51fa7448-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:18:33.183379 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.183281    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-577fz\" (UniqueName: \"kubernetes.io/projected/a66442e9-9658-48c7-abb9-1c7f51fa7448-kube-api-access-577fz\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:18:33.239107 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.239072    2557 generic.go:358] "Generic (PLEG): container finished" podID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerID="418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72" exitCode=0
May 20 14:18:33.239203 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.239109    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" event={"ID":"a66442e9-9658-48c7-abb9-1c7f51fa7448","Type":"ContainerDied","Data":"418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72"}
May 20 14:18:33.239203 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.239137    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr" event={"ID":"a66442e9-9658-48c7-abb9-1c7f51fa7448","Type":"ContainerDied","Data":"764a0d3defd02ead3d989cc62c9081195b20a1a5ba7d84c491e18005f542d7bc"}
May 20 14:18:33.239203 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.239157    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"
May 20 14:18:33.239328 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.239159    2557 scope.go:117] "RemoveContainer" containerID="0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636"
May 20 14:18:33.247055 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.247039    2557 scope.go:117] "RemoveContainer" containerID="418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72"
May 20 14:18:33.254244 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.254227    2557 scope.go:117] "RemoveContainer" containerID="95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68"
May 20 14:18:33.261094 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.261075    2557 scope.go:117] "RemoveContainer" containerID="0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636"
May 20 14:18:33.261405 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:18:33.261383    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636\": container with ID starting with 0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636 not found: ID does not exist" containerID="0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636"
May 20 14:18:33.261490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.261412    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636"} err="failed to get container status \"0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636\": rpc error: code = NotFound desc = could not find container \"0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636\": container with ID starting with 0a02c2c10293baf9bde73915f4bbdf2f5c8c91be80e395750575c4273826b636 not found: ID does not exist"
May 20 14:18:33.261490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.261432    2557 scope.go:117] "RemoveContainer" containerID="418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72"
May 20 14:18:33.261720 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:18:33.261698    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72\": container with ID starting with 418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72 not found: ID does not exist" containerID="418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72"
May 20 14:18:33.261789 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.261726    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72"} err="failed to get container status \"418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72\": rpc error: code = NotFound desc = could not find container \"418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72\": container with ID starting with 418fd928db71de0b06b18677bc0f3bb056a3d2bef45b6304d1da16294d309c72 not found: ID does not exist"
May 20 14:18:33.261789 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.261749    2557 scope.go:117] "RemoveContainer" containerID="95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68"
May 20 14:18:33.262256 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:18:33.262035    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68\": container with ID starting with 95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68 not found: ID does not exist" containerID="95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68"
May 20 14:18:33.262256 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.262065    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68"} err="failed to get container status \"95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68\": rpc error: code = NotFound desc = could not find container \"95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68\": container with ID starting with 95d679133a15ae977e1c7637438377a3160e6a6db1ee094f234ec7570d064e68 not found: ID does not exist"
May 20 14:18:33.263491 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.263469    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"]
May 20 14:18:33.271540 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:33.271521    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-fj7cr"]
May 20 14:18:34.404547 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:34.404510    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" path="/var/lib/kubelet/pods/a66442e9-9658-48c7-abb9-1c7f51fa7448/volumes"
May 20 14:18:36.251504 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:36.251469    2557 generic.go:358] "Generic (PLEG): container finished" podID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerID="d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79" exitCode=0
May 20 14:18:36.251923 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:36.251557    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" event={"ID":"33cc1b8e-0933-45e9-9999-1a1881d12d65","Type":"ContainerDied","Data":"d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79"}
May 20 14:18:37.257210 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:37.257173    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" event={"ID":"33cc1b8e-0933-45e9-9999-1a1881d12d65","Type":"ContainerStarted","Data":"d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed"}
May 20 14:18:37.257688 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:37.257218    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" event={"ID":"33cc1b8e-0933-45e9-9999-1a1881d12d65","Type":"ContainerStarted","Data":"ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f"}
May 20 14:18:37.257688 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:37.257459    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:37.278829 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:37.278786    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podStartSLOduration=7.278775389 podStartE2EDuration="7.278775389s" podCreationTimestamp="2026-05-20 14:18:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:18:37.278294856 +0000 UTC m=+1697.415849074" watchObservedRunningTime="2026-05-20 14:18:37.278775389 +0000 UTC m=+1697.416329606"
May 20 14:18:38.260613 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:38.260585    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:38.261774 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:38.261745    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 20 14:18:39.263552 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:39.263505    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 20 14:18:44.267636 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:44.267609    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:18:44.268264 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:44.268237    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 20 14:18:54.268586 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:18:54.268546    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 20 14:19:04.269087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:04.269052    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 20 14:19:14.268448 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:14.268411    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 20 14:19:24.269078 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:24.269041    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:19:32.218893 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.218844    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"]
May 20 14:19:32.219316 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.219268    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container" containerID="cri-o://ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f" gracePeriod=30
May 20 14:19:32.219395 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.219344    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kube-rbac-proxy" containerID="cri-o://d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed" gracePeriod=30
May 20 14:19:32.370594 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.370564    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"]
May 20 14:19:32.370956 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.370942    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kube-rbac-proxy"
May 20 14:19:32.371005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.370960    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kube-rbac-proxy"
May 20 14:19:32.371005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.370973    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container"
May 20 14:19:32.371005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.370978    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container"
May 20 14:19:32.371005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.370994    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="storage-initializer"
May 20 14:19:32.371005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.370999    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="storage-initializer"
May 20 14:19:32.371156 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.371056    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kserve-container"
May 20 14:19:32.371156 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.371070    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="a66442e9-9658-48c7-abb9-1c7f51fa7448" containerName="kube-rbac-proxy"
May 20 14:19:32.374130 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.374108    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.377184 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.377167    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
May 20 14:19:32.377184 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.377176    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
May 20 14:19:32.388477 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.388456    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"]
May 20 14:19:32.438824 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.438801    2557 generic.go:358] "Generic (PLEG): container finished" podID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerID="d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed" exitCode=2
May 20 14:19:32.438942 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.438876    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" event={"ID":"33cc1b8e-0933-45e9-9999-1a1881d12d65","Type":"ContainerDied","Data":"d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed"}
May 20 14:19:32.510465 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.510413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7999ab0-606a-4ad6-82e7-872484cd0536-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.510465 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.510438    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jxc\" (UniqueName: \"kubernetes.io/projected/f7999ab0-606a-4ad6-82e7-872484cd0536-kube-api-access-j2jxc\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.510577 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.510483    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7999ab0-606a-4ad6-82e7-872484cd0536-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.510577 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.510506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-cabundle-cert\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.510577 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.510525    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.611331 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.611305    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7999ab0-606a-4ad6-82e7-872484cd0536-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.611424 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.611335    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jxc\" (UniqueName: \"kubernetes.io/projected/f7999ab0-606a-4ad6-82e7-872484cd0536-kube-api-access-j2jxc\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.611424 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.611375    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7999ab0-606a-4ad6-82e7-872484cd0536-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.611424 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.611397    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-cabundle-cert\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.611564 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.611424    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.611831 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.611813    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7999ab0-606a-4ad6-82e7-872484cd0536-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.612091 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.612073    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-cabundle-cert\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.612132 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.612086    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.613528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.613511    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7999ab0-606a-4ad6-82e7-872484cd0536-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.622009 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.621990    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jxc\" (UniqueName: \"kubernetes.io/projected/f7999ab0-606a-4ad6-82e7-872484cd0536-kube-api-access-j2jxc\") pod \"isvc-pmml-predictor-8bb578669-hpbck\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.683605 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.683586    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:32.814482 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:32.814455    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"]
May 20 14:19:32.816150 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:19:32.816123    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7999ab0_606a_4ad6_82e7_872484cd0536.slice/crio-a736f5974675e8c43b8fc2ad033bdbb47188b8137fcb91d8b6799691a923a412 WatchSource:0}: Error finding container a736f5974675e8c43b8fc2ad033bdbb47188b8137fcb91d8b6799691a923a412: Status 404 returned error can't find the container with id a736f5974675e8c43b8fc2ad033bdbb47188b8137fcb91d8b6799691a923a412
May 20 14:19:33.443249 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:33.443215    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" event={"ID":"f7999ab0-606a-4ad6-82e7-872484cd0536","Type":"ContainerStarted","Data":"f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7"}
May 20 14:19:33.443249 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:33.443248    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" event={"ID":"f7999ab0-606a-4ad6-82e7-872484cd0536","Type":"ContainerStarted","Data":"a736f5974675e8c43b8fc2ad033bdbb47188b8137fcb91d8b6799691a923a412"}
May 20 14:19:34.263696 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.263654    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
May 20 14:19:34.268206 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.268175    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 20 14:19:34.665827 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.665803    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:19:34.730407 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.730381    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33cc1b8e-0933-45e9-9999-1a1881d12d65-kserve-provision-location\") pod \"33cc1b8e-0933-45e9-9999-1a1881d12d65\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") "
May 20 14:19:34.730533 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.730413    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cc1b8e-0933-45e9-9999-1a1881d12d65-proxy-tls\") pod \"33cc1b8e-0933-45e9-9999-1a1881d12d65\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") "
May 20 14:19:34.730533 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.730442    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"33cc1b8e-0933-45e9-9999-1a1881d12d65\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") "
May 20 14:19:34.730533 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.730516    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzsvc\" (UniqueName: \"kubernetes.io/projected/33cc1b8e-0933-45e9-9999-1a1881d12d65-kube-api-access-qzsvc\") pod \"33cc1b8e-0933-45e9-9999-1a1881d12d65\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") "
May 20 14:19:34.730533 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.730533    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-cabundle-cert\") pod \"33cc1b8e-0933-45e9-9999-1a1881d12d65\" (UID: \"33cc1b8e-0933-45e9-9999-1a1881d12d65\") "
May 20 14:19:34.730873 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.730829    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "33cc1b8e-0933-45e9-9999-1a1881d12d65" (UID: "33cc1b8e-0933-45e9-9999-1a1881d12d65"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:19:34.730952 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.730937    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33cc1b8e-0933-45e9-9999-1a1881d12d65" (UID: "33cc1b8e-0933-45e9-9999-1a1881d12d65"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:19:34.732415 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.732389    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33cc1b8e-0933-45e9-9999-1a1881d12d65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33cc1b8e-0933-45e9-9999-1a1881d12d65" (UID: "33cc1b8e-0933-45e9-9999-1a1881d12d65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:19:34.732513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.732493    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33cc1b8e-0933-45e9-9999-1a1881d12d65-kube-api-access-qzsvc" (OuterVolumeSpecName: "kube-api-access-qzsvc") pod "33cc1b8e-0933-45e9-9999-1a1881d12d65" (UID: "33cc1b8e-0933-45e9-9999-1a1881d12d65"). InnerVolumeSpecName "kube-api-access-qzsvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:19:34.739843 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.739816    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33cc1b8e-0933-45e9-9999-1a1881d12d65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33cc1b8e-0933-45e9-9999-1a1881d12d65" (UID: "33cc1b8e-0933-45e9-9999-1a1881d12d65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:19:34.831706 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.831686    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzsvc\" (UniqueName: \"kubernetes.io/projected/33cc1b8e-0933-45e9-9999-1a1881d12d65-kube-api-access-qzsvc\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:19:34.831778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.831707    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:19:34.831778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.831718    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33cc1b8e-0933-45e9-9999-1a1881d12d65-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:19:34.831778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.831726    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cc1b8e-0933-45e9-9999-1a1881d12d65-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:19:34.831778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:34.831736    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cc1b8e-0933-45e9-9999-1a1881d12d65-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:19:35.450995 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.450961    2557 generic.go:358] "Generic (PLEG): container finished" podID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerID="ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f" exitCode=0
May 20 14:19:35.451108 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.451010    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" event={"ID":"33cc1b8e-0933-45e9-9999-1a1881d12d65","Type":"ContainerDied","Data":"ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f"}
May 20 14:19:35.451108 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.451031    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc" event={"ID":"33cc1b8e-0933-45e9-9999-1a1881d12d65","Type":"ContainerDied","Data":"713c59d88da8b55487d21bd656b6edf0346646c40ac9d6295355c05fbb02c823"}
May 20 14:19:35.451108 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.451048    2557 scope.go:117] "RemoveContainer" containerID="d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed"
May 20 14:19:35.451108 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.451067    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"
May 20 14:19:35.460172 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.460143    2557 scope.go:117] "RemoveContainer" containerID="ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f"
May 20 14:19:35.467984 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.467964    2557 scope.go:117] "RemoveContainer" containerID="d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79"
May 20 14:19:35.475432 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.475411    2557 scope.go:117] "RemoveContainer" containerID="d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed"
May 20 14:19:35.475728 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:19:35.475705    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed\": container with ID starting with d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed not found: ID does not exist" containerID="d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed"
May 20 14:19:35.475825 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.475734    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed"} err="failed to get container status \"d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed\": rpc error: code = NotFound desc = could not find container \"d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed\": container with ID starting with d17a1dbd7085ae45d9ce6581950f35de9531db03efe3ea0044483bd2c86dffed not found: ID does not exist"
May 20 14:19:35.475825 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.475754    2557 scope.go:117] "RemoveContainer" containerID="ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f"
May 20 14:19:35.476070 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:19:35.476051    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f\": container with ID starting with ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f not found: ID does not exist" containerID="ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f"
May 20 14:19:35.476118 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.476076    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f"} err="failed to get container status \"ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f\": rpc error: code = NotFound desc = could not find container \"ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f\": container with ID starting with ee6aee4aec02e8e670156301a25fd5edbb70fead7c53262c77e6de8cebd0cc9f not found: ID does not exist"
May 20 14:19:35.476118 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.476104    2557 scope.go:117] "RemoveContainer" containerID="d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79"
May 20 14:19:35.476395 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:19:35.476377    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79\": container with ID starting with d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79 not found: ID does not exist" containerID="d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79"
May 20 14:19:35.476446 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.476403    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79"} err="failed to get container status \"d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79\": rpc error: code = NotFound desc = could not find container \"d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79\": container with ID starting with d8dc449a226dc4364ef3078d6118012270fe15ea5e7699a4496329ea61831f79 not found: ID does not exist"
May 20 14:19:35.481832 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.481808    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"]
May 20 14:19:35.484344 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:35.484322    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-sr5tc"]
May 20 14:19:36.405317 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:36.405273    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" path="/var/lib/kubelet/pods/33cc1b8e-0933-45e9-9999-1a1881d12d65/volumes"
May 20 14:19:36.458073 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:36.458046    2557 generic.go:358] "Generic (PLEG): container finished" podID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerID="f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7" exitCode=0
May 20 14:19:36.458180 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:36.458123    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" event={"ID":"f7999ab0-606a-4ad6-82e7-872484cd0536","Type":"ContainerDied","Data":"f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7"}
May 20 14:19:43.486183 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:43.486149    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" event={"ID":"f7999ab0-606a-4ad6-82e7-872484cd0536","Type":"ContainerStarted","Data":"dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2"}
May 20 14:19:44.491328 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:44.491290    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" event={"ID":"f7999ab0-606a-4ad6-82e7-872484cd0536","Type":"ContainerStarted","Data":"8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5"}
May 20 14:19:44.491754 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:44.491458    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:44.513987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:44.513940    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podStartSLOduration=5.557984739 podStartE2EDuration="12.513928359s" podCreationTimestamp="2026-05-20 14:19:32 +0000 UTC" firstStartedPulling="2026-05-20 14:19:36.459264606 +0000 UTC m=+1756.596818802" lastFinishedPulling="2026-05-20 14:19:43.415208226 +0000 UTC m=+1763.552762422" observedRunningTime="2026-05-20 14:19:44.512039943 +0000 UTC m=+1764.649594170" watchObservedRunningTime="2026-05-20 14:19:44.513928359 +0000 UTC m=+1764.651482642"
May 20 14:19:45.495090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:45.495062    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:45.496404 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:45.496375    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:19:46.498481 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:46.498444    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:19:51.504490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:51.504388    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:19:51.505086 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:19:51.505058    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:20:01.505016 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:01.504935    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:20:11.505014 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:11.504973    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:20:20.431690 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:20.431663    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:20:20.433148 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:20.433123    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:20:20.435919 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:20.435898    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:20:20.437423 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:20.437403    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:20:21.505113 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:21.505074    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:20:31.505692 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:31.505655    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:20:41.505103 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:41.505059    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:20:51.505088 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:51.505045    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:20:54.401785 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:20:54.401738    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:21:04.405169 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:04.405129    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:21:13.394263 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.394229    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"]
May 20 14:21:13.394948 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.394548    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" containerID="cri-o://dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2" gracePeriod=30
May 20 14:21:13.394948 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.394665    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kube-rbac-proxy" containerID="cri-o://8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5" gracePeriod=30
May 20 14:21:13.501850 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.501813    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"]
May 20 14:21:13.502285 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.502265    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="storage-initializer"
May 20 14:21:13.502285 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.502283    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="storage-initializer"
May 20 14:21:13.502448 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.502295    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kube-rbac-proxy"
May 20 14:21:13.502448 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.502301    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kube-rbac-proxy"
May 20 14:21:13.502448 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.502310    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container"
May 20 14:21:13.502448 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.502315    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container"
May 20 14:21:13.502448 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.502381    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kserve-container"
May 20 14:21:13.502448 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.502390    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="33cc1b8e-0933-45e9-9999-1a1881d12d65" containerName="kube-rbac-proxy"
May 20 14:21:13.505547 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.505529    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.508273 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.508252    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
May 20 14:21:13.508376 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.508273    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
May 20 14:21:13.514918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.514897    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"]
May 20 14:21:13.665946 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.665890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.666040 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.665945    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rm5r\" (UniqueName: \"kubernetes.io/projected/d5b0085a-24c4-4539-b60c-e303df4469ff-kube-api-access-9rm5r\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.666040 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.665981    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.666040 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.666013    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b0085a-24c4-4539-b60c-e303df4469ff-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.666161 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.666051    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b0085a-24c4-4539-b60c-e303df4469ff-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.766531 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.766502    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9rm5r\" (UniqueName: \"kubernetes.io/projected/d5b0085a-24c4-4539-b60c-e303df4469ff-kube-api-access-9rm5r\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.766635 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.766539    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.766635 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.766580    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b0085a-24c4-4539-b60c-e303df4469ff-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.766635 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.766609    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b0085a-24c4-4539-b60c-e303df4469ff-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.766825 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.766673    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.767122 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.767096    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b0085a-24c4-4539-b60c-e303df4469ff-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.767310 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.767290    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.767385 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.767320    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.769213 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.769192    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b0085a-24c4-4539-b60c-e303df4469ff-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.770547 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.770524    2557 generic.go:358] "Generic (PLEG): container finished" podID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerID="8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5" exitCode=2
May 20 14:21:13.770637 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.770592    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" event={"ID":"f7999ab0-606a-4ad6-82e7-872484cd0536","Type":"ContainerDied","Data":"8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5"}
May 20 14:21:13.776183 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.776165    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rm5r\" (UniqueName: \"kubernetes.io/projected/d5b0085a-24c4-4539-b60c-e303df4469ff-kube-api-access-9rm5r\") pod \"isvc-pmml-runtime-predictor-67bc544947-989xz\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.816137 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.816112    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:13.958637 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:13.958570    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"]
May 20 14:21:13.961741 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:21:13.961716    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b0085a_24c4_4539_b60c_e303df4469ff.slice/crio-9b0fa6c2e6b9d31d9d59bf8a9ffdcbe5592e6af28c56dc8285839773ee32d0fb WatchSource:0}: Error finding container 9b0fa6c2e6b9d31d9d59bf8a9ffdcbe5592e6af28c56dc8285839773ee32d0fb: Status 404 returned error can't find the container with id 9b0fa6c2e6b9d31d9d59bf8a9ffdcbe5592e6af28c56dc8285839773ee32d0fb
May 20 14:21:14.402351 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:14.402313    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 20 14:21:14.775638 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:14.775559    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" event={"ID":"d5b0085a-24c4-4539-b60c-e303df4469ff","Type":"ContainerStarted","Data":"fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66"}
May 20 14:21:14.775638 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:14.775596    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" event={"ID":"d5b0085a-24c4-4539-b60c-e303df4469ff","Type":"ContainerStarted","Data":"9b0fa6c2e6b9d31d9d59bf8a9ffdcbe5592e6af28c56dc8285839773ee32d0fb"}
May 20 14:21:16.444230 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.444209    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:21:16.588547 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.588527    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"f7999ab0-606a-4ad6-82e7-872484cd0536\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") "
May 20 14:21:16.588667 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.588567    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-cabundle-cert\") pod \"f7999ab0-606a-4ad6-82e7-872484cd0536\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") "
May 20 14:21:16.588667 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.588632    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7999ab0-606a-4ad6-82e7-872484cd0536-kserve-provision-location\") pod \"f7999ab0-606a-4ad6-82e7-872484cd0536\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") "
May 20 14:21:16.588776 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.588675    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7999ab0-606a-4ad6-82e7-872484cd0536-proxy-tls\") pod \"f7999ab0-606a-4ad6-82e7-872484cd0536\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") "
May 20 14:21:16.588776 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.588708    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2jxc\" (UniqueName: \"kubernetes.io/projected/f7999ab0-606a-4ad6-82e7-872484cd0536-kube-api-access-j2jxc\") pod \"f7999ab0-606a-4ad6-82e7-872484cd0536\" (UID: \"f7999ab0-606a-4ad6-82e7-872484cd0536\") "
May 20 14:21:16.588972 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.588946    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "f7999ab0-606a-4ad6-82e7-872484cd0536" (UID: "f7999ab0-606a-4ad6-82e7-872484cd0536"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:21:16.589040 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.589013    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7999ab0-606a-4ad6-82e7-872484cd0536-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7999ab0-606a-4ad6-82e7-872484cd0536" (UID: "f7999ab0-606a-4ad6-82e7-872484cd0536"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:21:16.589040 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.589027    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7999ab0-606a-4ad6-82e7-872484cd0536" (UID: "f7999ab0-606a-4ad6-82e7-872484cd0536"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:21:16.590730 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.590710    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7999ab0-606a-4ad6-82e7-872484cd0536-kube-api-access-j2jxc" (OuterVolumeSpecName: "kube-api-access-j2jxc") pod "f7999ab0-606a-4ad6-82e7-872484cd0536" (UID: "f7999ab0-606a-4ad6-82e7-872484cd0536"). InnerVolumeSpecName "kube-api-access-j2jxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:21:16.590774 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.590735    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7999ab0-606a-4ad6-82e7-872484cd0536-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7999ab0-606a-4ad6-82e7-872484cd0536" (UID: "f7999ab0-606a-4ad6-82e7-872484cd0536"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:21:16.689447 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.689419    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7999ab0-606a-4ad6-82e7-872484cd0536-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:21:16.689447 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.689443    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jxc\" (UniqueName: \"kubernetes.io/projected/f7999ab0-606a-4ad6-82e7-872484cd0536-kube-api-access-j2jxc\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:21:16.689593 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.689459    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:21:16.689593 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.689473    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7999ab0-606a-4ad6-82e7-872484cd0536-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:21:16.689593 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.689487    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7999ab0-606a-4ad6-82e7-872484cd0536-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:21:16.783955 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.783926    2557 generic.go:358] "Generic (PLEG): container finished" podID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerID="dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2" exitCode=0
May 20 14:21:16.784064 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.784007    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"
May 20 14:21:16.784064 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.784013    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" event={"ID":"f7999ab0-606a-4ad6-82e7-872484cd0536","Type":"ContainerDied","Data":"dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2"}
May 20 14:21:16.784064 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.784056    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck" event={"ID":"f7999ab0-606a-4ad6-82e7-872484cd0536","Type":"ContainerDied","Data":"a736f5974675e8c43b8fc2ad033bdbb47188b8137fcb91d8b6799691a923a412"}
May 20 14:21:16.784168 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.784073    2557 scope.go:117] "RemoveContainer" containerID="8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5"
May 20 14:21:16.794124 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.794015    2557 scope.go:117] "RemoveContainer" containerID="dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2"
May 20 14:21:16.802082 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.802063    2557 scope.go:117] "RemoveContainer" containerID="f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7"
May 20 14:21:16.809676 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.809650    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"]
May 20 14:21:16.809846 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.809794    2557 scope.go:117] "RemoveContainer" containerID="8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5"
May 20 14:21:16.810061 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:21:16.810043    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5\": container with ID starting with 8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5 not found: ID does not exist" containerID="8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5"
May 20 14:21:16.810116 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.810068    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5"} err="failed to get container status \"8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5\": rpc error: code = NotFound desc = could not find container \"8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5\": container with ID starting with 8e96a947e66523d8bd5f030a59dec8bff185c983d51bd18aedcf6701045faff5 not found: ID does not exist"
May 20 14:21:16.810116 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.810084    2557 scope.go:117] "RemoveContainer" containerID="dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2"
May 20 14:21:16.810322 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:21:16.810307    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2\": container with ID starting with dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2 not found: ID does not exist" containerID="dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2"
May 20 14:21:16.810362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.810325    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2"} err="failed to get container status \"dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2\": rpc error: code = NotFound desc = could not find container \"dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2\": container with ID starting with dc93e5f4370891f030482090c25149958485e3934437f7b21fba82947ea29dd2 not found: ID does not exist"
May 20 14:21:16.810362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.810341    2557 scope.go:117] "RemoveContainer" containerID="f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7"
May 20 14:21:16.810562 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:21:16.810545    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7\": container with ID starting with f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7 not found: ID does not exist" containerID="f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7"
May 20 14:21:16.810601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.810568    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7"} err="failed to get container status \"f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7\": rpc error: code = NotFound desc = could not find container \"f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7\": container with ID starting with f6c337c389f6ce5155329d7c4471f63b0d0684e9c432b3f923d1801f8f8b57c7 not found: ID does not exist"
May 20 14:21:16.814077 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:16.814057    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-hpbck"]
May 20 14:21:17.788559 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:17.788527    2557 generic.go:358] "Generic (PLEG): container finished" podID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerID="fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66" exitCode=0
May 20 14:21:17.788937 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:17.788604    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" event={"ID":"d5b0085a-24c4-4539-b60c-e303df4469ff","Type":"ContainerDied","Data":"fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66"}
May 20 14:21:18.405556 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:18.405520    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" path="/var/lib/kubelet/pods/f7999ab0-606a-4ad6-82e7-872484cd0536/volumes"
May 20 14:21:18.795361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:18.795281    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" event={"ID":"d5b0085a-24c4-4539-b60c-e303df4469ff","Type":"ContainerStarted","Data":"28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b"}
May 20 14:21:18.795361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:18.795326    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" event={"ID":"d5b0085a-24c4-4539-b60c-e303df4469ff","Type":"ContainerStarted","Data":"2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5"}
May 20 14:21:18.795767 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:18.795692    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:18.795821 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:18.795800    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:18.796919 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:18.796884    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:21:18.821539 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:18.821487    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podStartSLOduration=5.821472464 podStartE2EDuration="5.821472464s" podCreationTimestamp="2026-05-20 14:21:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:21:18.819668117 +0000 UTC m=+1858.957222335" watchObservedRunningTime="2026-05-20 14:21:18.821472464 +0000 UTC m=+1858.959026683"
May 20 14:21:19.798942 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:19.798896    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:21:24.803379 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:24.803351    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:21:24.805806 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:24.803954    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:21:34.803986 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:34.803905    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:21:44.803947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:44.803905    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:21:54.803897 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:21:54.803849    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:22:04.804240 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:04.804199    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:22:14.804387 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:14.804303    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:22:24.804250 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:24.804211    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:22:30.403341 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:30.403299    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 20 14:22:40.405714 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:40.405687    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:22:44.612265 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.612225    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"]
May 20 14:22:44.612728 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.612537    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container" containerID="cri-o://2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5" gracePeriod=30
May 20 14:22:44.612728 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.612581    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kube-rbac-proxy" containerID="cri-o://28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b" gracePeriod=30
May 20 14:22:44.799517 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.799484    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
May 20 14:22:44.922172 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.922117    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"]
May 20 14:22:44.922469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.922458    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kube-rbac-proxy"
May 20 14:22:44.922513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.922471    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kube-rbac-proxy"
May 20 14:22:44.922513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.922487    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container"
May 20 14:22:44.922513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.922493    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container"
May 20 14:22:44.922618 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.922516    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="storage-initializer"
May 20 14:22:44.922618 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.922522    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="storage-initializer"
May 20 14:22:44.922618 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.922571    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kserve-container"
May 20 14:22:44.922618 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.922600    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f7999ab0-606a-4ad6-82e7-872484cd0536" containerName="kube-rbac-proxy"
May 20 14:22:44.925618 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.925604    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:44.932413 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.932399    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
May 20 14:22:44.933293 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.933277    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
May 20 14:22:44.941008 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:44.940989    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"]
May 20 14:22:45.028934 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.028909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.029027 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.028941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.029027 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.028976    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.029106 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.029041    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.029106 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.029079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcczc\" (UniqueName: \"kubernetes.io/projected/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kube-api-access-wcczc\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.086837 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.086811    2557 generic.go:358] "Generic (PLEG): container finished" podID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerID="28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b" exitCode=2
May 20 14:22:45.086938 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.086894    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" event={"ID":"d5b0085a-24c4-4539-b60c-e303df4469ff","Type":"ContainerDied","Data":"28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b"}
May 20 14:22:45.129756 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.129734    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.129844 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.129759    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.129844 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.129790    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.129844 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.129838    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.129975 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.129891    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wcczc\" (UniqueName: \"kubernetes.io/projected/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kube-api-access-wcczc\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.130240 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.130218    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.130439 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.130425    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.130497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.130479    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.132130 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.132113    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.148086 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.148059    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcczc\" (UniqueName: \"kubernetes.io/projected/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kube-api-access-wcczc\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.235506 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.235453    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:45.364903 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:45.364847    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"]
May 20 14:22:45.367139 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:22:45.367116    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdee6b7b_d8e1_49a5_ba10_3998b4dfe61e.slice/crio-1947e0b343af4c6cade3c76b1d4c151152162ae7889ce6ddeaa1480b1e435a06 WatchSource:0}: Error finding container 1947e0b343af4c6cade3c76b1d4c151152162ae7889ce6ddeaa1480b1e435a06: Status 404 returned error can't find the container with id 1947e0b343af4c6cade3c76b1d4c151152162ae7889ce6ddeaa1480b1e435a06
May 20 14:22:46.092241 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:46.092201    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" event={"ID":"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e","Type":"ContainerStarted","Data":"e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b"}
May 20 14:22:46.092595 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:46.092250    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" event={"ID":"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e","Type":"ContainerStarted","Data":"1947e0b343af4c6cade3c76b1d4c151152162ae7889ce6ddeaa1480b1e435a06"}
May 20 14:22:47.744907 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.744885    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:22:47.852614 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.852550    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"d5b0085a-24c4-4539-b60c-e303df4469ff\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") "
May 20 14:22:47.852614 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.852592    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b0085a-24c4-4539-b60c-e303df4469ff-proxy-tls\") pod \"d5b0085a-24c4-4539-b60c-e303df4469ff\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") "
May 20 14:22:47.852791 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.852645    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b0085a-24c4-4539-b60c-e303df4469ff-kserve-provision-location\") pod \"d5b0085a-24c4-4539-b60c-e303df4469ff\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") "
May 20 14:22:47.852791 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.852711    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rm5r\" (UniqueName: \"kubernetes.io/projected/d5b0085a-24c4-4539-b60c-e303df4469ff-kube-api-access-9rm5r\") pod \"d5b0085a-24c4-4539-b60c-e303df4469ff\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") "
May 20 14:22:47.852791 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.852727    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-cabundle-cert\") pod \"d5b0085a-24c4-4539-b60c-e303df4469ff\" (UID: \"d5b0085a-24c4-4539-b60c-e303df4469ff\") "
May 20 14:22:47.852988 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.852889    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "d5b0085a-24c4-4539-b60c-e303df4469ff" (UID: "d5b0085a-24c4-4539-b60c-e303df4469ff"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:22:47.853086 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.853058    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b0085a-24c4-4539-b60c-e303df4469ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5b0085a-24c4-4539-b60c-e303df4469ff" (UID: "d5b0085a-24c4-4539-b60c-e303df4469ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:47.853160 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.853126    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5b0085a-24c4-4539-b60c-e303df4469ff" (UID: "d5b0085a-24c4-4539-b60c-e303df4469ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:22:47.854802 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.854784    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b0085a-24c4-4539-b60c-e303df4469ff-kube-api-access-9rm5r" (OuterVolumeSpecName: "kube-api-access-9rm5r") pod "d5b0085a-24c4-4539-b60c-e303df4469ff" (UID: "d5b0085a-24c4-4539-b60c-e303df4469ff"). InnerVolumeSpecName "kube-api-access-9rm5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:22:47.854882 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.854848    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b0085a-24c4-4539-b60c-e303df4469ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5b0085a-24c4-4539-b60c-e303df4469ff" (UID: "d5b0085a-24c4-4539-b60c-e303df4469ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:22:47.953712 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.953691    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b0085a-24c4-4539-b60c-e303df4469ff-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:22:47.953712 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.953712    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b0085a-24c4-4539-b60c-e303df4469ff-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:22:47.953841 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.953721    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rm5r\" (UniqueName: \"kubernetes.io/projected/d5b0085a-24c4-4539-b60c-e303df4469ff-kube-api-access-9rm5r\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:22:47.953841 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.953732    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:22:47.953841 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:47.953741    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b0085a-24c4-4539-b60c-e303df4469ff-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:22:48.103187 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.103127    2557 generic.go:358] "Generic (PLEG): container finished" podID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerID="2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5" exitCode=0
May 20 14:22:48.103293 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.103214    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"
May 20 14:22:48.103358 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.103211    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" event={"ID":"d5b0085a-24c4-4539-b60c-e303df4469ff","Type":"ContainerDied","Data":"2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5"}
May 20 14:22:48.103358 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.103321    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz" event={"ID":"d5b0085a-24c4-4539-b60c-e303df4469ff","Type":"ContainerDied","Data":"9b0fa6c2e6b9d31d9d59bf8a9ffdcbe5592e6af28c56dc8285839773ee32d0fb"}
May 20 14:22:48.103358 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.103337    2557 scope.go:117] "RemoveContainer" containerID="28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b"
May 20 14:22:48.111493 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.111475    2557 scope.go:117] "RemoveContainer" containerID="2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5"
May 20 14:22:48.118371 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.118357    2557 scope.go:117] "RemoveContainer" containerID="fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66"
May 20 14:22:48.125384 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.125364    2557 scope.go:117] "RemoveContainer" containerID="28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b"
May 20 14:22:48.125649 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:22:48.125630    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b\": container with ID starting with 28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b not found: ID does not exist" containerID="28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b"
May 20 14:22:48.125712 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.125657    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b"} err="failed to get container status \"28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b\": rpc error: code = NotFound desc = could not find container \"28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b\": container with ID starting with 28ba2c07affb17837cb4fd48895db14f68cb71f9af6f940c4cf10b1608eaed5b not found: ID does not exist"
May 20 14:22:48.125712 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.125674    2557 scope.go:117] "RemoveContainer" containerID="2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5"
May 20 14:22:48.126010 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:22:48.125959    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5\": container with ID starting with 2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5 not found: ID does not exist" containerID="2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5"
May 20 14:22:48.126094 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.126018    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5"} err="failed to get container status \"2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5\": rpc error: code = NotFound desc = could not find container \"2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5\": container with ID starting with 2500269395c0969273aba74c1defa5d2310de0102b4389b8cf512dfa3521e5c5 not found: ID does not exist"
May 20 14:22:48.126094 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.126046    2557 scope.go:117] "RemoveContainer" containerID="fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66"
May 20 14:22:48.126094 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.126090    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"]
May 20 14:22:48.126306 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:22:48.126288    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66\": container with ID starting with fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66 not found: ID does not exist" containerID="fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66"
May 20 14:22:48.126361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.126312    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66"} err="failed to get container status \"fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66\": rpc error: code = NotFound desc = could not find container \"fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66\": container with ID starting with fdeea8892141c483a079b510c6881d53de3d502750652560e851ae2379702b66 not found: ID does not exist"
May 20 14:22:48.128298 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.128279    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-989xz"]
May 20 14:22:48.405816 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:48.405754    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" path="/var/lib/kubelet/pods/d5b0085a-24c4-4539-b60c-e303df4469ff/volumes"
May 20 14:22:50.113506 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:50.113470    2557 generic.go:358] "Generic (PLEG): container finished" podID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerID="e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b" exitCode=0
May 20 14:22:50.113998 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:50.113519    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" event={"ID":"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e","Type":"ContainerDied","Data":"e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b"}
May 20 14:22:51.118754 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:51.118715    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" event={"ID":"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e","Type":"ContainerStarted","Data":"d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a"}
May 20 14:22:51.118754 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:51.118759    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" event={"ID":"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e","Type":"ContainerStarted","Data":"f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def"}
May 20 14:22:51.119226 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:51.118997    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:51.144253 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:51.144209    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podStartSLOduration=7.144196486 podStartE2EDuration="7.144196486s" podCreationTimestamp="2026-05-20 14:22:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:22:51.141987322 +0000 UTC m=+1951.279541559" watchObservedRunningTime="2026-05-20 14:22:51.144196486 +0000 UTC m=+1951.281750704"
May 20 14:22:52.121429 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:52.121398    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:52.122548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:52.122523    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:22:53.124827 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:53.124787    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:22:58.129247 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:58.129217    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:22:58.129807 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:22:58.129781    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:23:08.129876 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:23:08.129769    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:23:18.130579 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:23:18.130540    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:23:28.129827 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:23:28.129786    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:23:38.130276 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:23:38.130234    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:23:48.130058 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:23:48.130018    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:23:58.130048 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:23:58.130006    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:24:08.130495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:08.130462    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:24:15.772469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.772435    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"]
May 20 14:24:15.772927 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.772743    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" containerID="cri-o://f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def" gracePeriod=30
May 20 14:24:15.772927 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.772791    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kube-rbac-proxy" containerID="cri-o://d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a" gracePeriod=30
May 20 14:24:15.985341 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.985313    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"]
May 20 14:24:15.985663 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.985651    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container"
May 20 14:24:15.985707 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.985664    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container"
May 20 14:24:15.985707 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.985675    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kube-rbac-proxy"
May 20 14:24:15.985707 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.985681    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kube-rbac-proxy"
May 20 14:24:15.985707 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.985696    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="storage-initializer"
May 20 14:24:15.985707 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.985701    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="storage-initializer"
May 20 14:24:15.985882 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.985758    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kserve-container"
May 20 14:24:15.985882 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.985765    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="d5b0085a-24c4-4539-b60c-e303df4469ff" containerName="kube-rbac-proxy"
May 20 14:24:15.988834 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.988810    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:15.995082 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:15.995051    2557 status_manager.go:895] "Failed to get status for pod" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" err="pods \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object"
May 20 14:24:15.995191 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:24:15.995167    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-primary-9eb24d-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9eb24d-predictor-serving-cert\"" type="*v1.Secret"
May 20 14:24:15.995253 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:24:15.995223    2557 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-primary-9eb24d-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9eb24d-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
May 20 14:24:16.020487 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.020464    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"]
May 20 14:24:16.071364 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.071344    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-cabundle-cert\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.071453 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.071374    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-996vq\" (UniqueName: \"kubernetes.io/projected/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kube-api-access-996vq\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.071453 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.071410    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kserve-provision-location\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.071552 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.071470    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-isvc-primary-9eb24d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.071552 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.071494    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-proxy-tls\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.172677 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.172649    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-cabundle-cert\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.172788 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.172689    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-996vq\" (UniqueName: \"kubernetes.io/projected/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kube-api-access-996vq\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.172788 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.172742    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kserve-provision-location\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.172942 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.172789    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-isvc-primary-9eb24d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.172942 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.172822    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-proxy-tls\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.173149 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.173127    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kserve-provision-location\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.173363 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.173346    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-cabundle-cert\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.182681 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.182655    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-996vq\" (UniqueName: \"kubernetes.io/projected/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kube-api-access-996vq\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.402346 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.402270    2557 generic.go:358] "Generic (PLEG): container finished" podID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerID="d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a" exitCode=2
May 20 14:24:16.404879 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.404829    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" event={"ID":"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e","Type":"ContainerDied","Data":"d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a"}
May 20 14:24:16.866226 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.866205    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9eb24d-predictor-serving-cert\""
May 20 14:24:16.876164 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.876138    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-proxy-tls\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:16.902446 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.902424    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9eb24d-kube-rbac-proxy-sar-config\""
May 20 14:24:16.903817 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:16.903797    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-isvc-primary-9eb24d-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9eb24d-predictor-86f8cd785d-wslfn\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") " pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:17.198484 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:17.198412    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:17.324392 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:17.324361    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"]
May 20 14:24:17.328621 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:24:17.328583    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6256f6d6_f1fd_42e5_847b_d8e85faf6c0f.slice/crio-1273cc173dcb32765935737b325708ed3f48e38736b2a4651008855fff5d1e0f WatchSource:0}: Error finding container 1273cc173dcb32765935737b325708ed3f48e38736b2a4651008855fff5d1e0f: Status 404 returned error can't find the container with id 1273cc173dcb32765935737b325708ed3f48e38736b2a4651008855fff5d1e0f
May 20 14:24:17.330514 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:17.330495    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:24:17.408124 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:17.408064    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" event={"ID":"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f","Type":"ContainerStarted","Data":"a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88"}
May 20 14:24:17.408124 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:17.408100    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" event={"ID":"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f","Type":"ContainerStarted","Data":"1273cc173dcb32765935737b325708ed3f48e38736b2a4651008855fff5d1e0f"}
May 20 14:24:18.125199 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.125150    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 20 14:24:18.130491 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.130464    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 20 14:24:18.713538 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.713515    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:24:18.793756 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.793698    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-cabundle-cert\") pod \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") "
May 20 14:24:18.793890 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.793779    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kserve-provision-location\") pod \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") "
May 20 14:24:18.793890 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.793809    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcczc\" (UniqueName: \"kubernetes.io/projected/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kube-api-access-wcczc\") pod \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") "
May 20 14:24:18.793890 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.793829    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") "
May 20 14:24:18.793890 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.793849    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-proxy-tls\") pod \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\" (UID: \"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e\") "
May 20 14:24:18.794145 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.794120    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" (UID: "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:24:18.794208 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.794185    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" (UID: "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:24:18.794307 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.794284    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" (UID: "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:24:18.795803 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.795773    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kube-api-access-wcczc" (OuterVolumeSpecName: "kube-api-access-wcczc") pod "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" (UID: "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e"). InnerVolumeSpecName "kube-api-access-wcczc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:24:18.795920 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.795797    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" (UID: "bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:24:18.894523 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.894489    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:24:18.894523 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.894522    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcczc\" (UniqueName: \"kubernetes.io/projected/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-kube-api-access-wcczc\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:24:18.894653 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.894533    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:24:18.894653 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.894543    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:24:18.894653 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:18.894552    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:24:19.416625 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.416596    2557 generic.go:358] "Generic (PLEG): container finished" podID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerID="f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def" exitCode=0
May 20 14:24:19.416991 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.416669    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" event={"ID":"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e","Type":"ContainerDied","Data":"f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def"}
May 20 14:24:19.416991 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.416693    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg" event={"ID":"bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e","Type":"ContainerDied","Data":"1947e0b343af4c6cade3c76b1d4c151152162ae7889ce6ddeaa1480b1e435a06"}
May 20 14:24:19.416991 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.416708    2557 scope.go:117] "RemoveContainer" containerID="d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a"
May 20 14:24:19.416991 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.416671    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"
May 20 14:24:19.425973 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.425954    2557 scope.go:117] "RemoveContainer" containerID="f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def"
May 20 14:24:19.433675 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.433655    2557 scope.go:117] "RemoveContainer" containerID="e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b"
May 20 14:24:19.441368 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.441343    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"]
May 20 14:24:19.442017 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.441993    2557 scope.go:117] "RemoveContainer" containerID="d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a"
May 20 14:24:19.442296 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:24:19.442280    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a\": container with ID starting with d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a not found: ID does not exist" containerID="d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a"
May 20 14:24:19.442363 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.442303    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a"} err="failed to get container status \"d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a\": rpc error: code = NotFound desc = could not find container \"d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a\": container with ID starting with d7df8f39e4ebd58310babf47da56b772cde44b3b6ef77763ef950bd78c5d274a not found: ID does not exist"
May 20 14:24:19.442363 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.442320    2557 scope.go:117] "RemoveContainer" containerID="f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def"
May 20 14:24:19.442592 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:24:19.442563    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def\": container with ID starting with f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def not found: ID does not exist" containerID="f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def"
May 20 14:24:19.442633 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.442597    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def"} err="failed to get container status \"f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def\": rpc error: code = NotFound desc = could not find container \"f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def\": container with ID starting with f71ab086b31845eceb362539d493e14cb6c56ea613a71f2acdba683109c11def not found: ID does not exist"
May 20 14:24:19.442633 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.442616    2557 scope.go:117] "RemoveContainer" containerID="e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b"
May 20 14:24:19.442908 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:24:19.442841    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b\": container with ID starting with e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b not found: ID does not exist" containerID="e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b"
May 20 14:24:19.442908 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.442879    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b"} err="failed to get container status \"e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b\": rpc error: code = NotFound desc = could not find container \"e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b\": container with ID starting with e98921f807a49013d6bdc6ad61549aaf327a28a413563314247c409289a0a58b not found: ID does not exist"
May 20 14:24:19.449866 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:19.449828    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-6zqkg"]
May 20 14:24:20.406193 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:20.406162    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" path="/var/lib/kubelet/pods/bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e/volumes"
May 20 14:24:21.425258 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:21.425229    2557 generic.go:358] "Generic (PLEG): container finished" podID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerID="a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88" exitCode=0
May 20 14:24:21.425579 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:21.425303    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" event={"ID":"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f","Type":"ContainerDied","Data":"a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88"}
May 20 14:24:22.430585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:22.430546    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" event={"ID":"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f","Type":"ContainerStarted","Data":"84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b"}
May 20 14:24:22.430585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:22.430585    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" event={"ID":"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f","Type":"ContainerStarted","Data":"da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f"}
May 20 14:24:22.431040 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:22.430792    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:22.458360 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:22.458315    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podStartSLOduration=7.458302368 podStartE2EDuration="7.458302368s" podCreationTimestamp="2026-05-20 14:24:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:24:22.455294743 +0000 UTC m=+2042.592848960" watchObservedRunningTime="2026-05-20 14:24:22.458302368 +0000 UTC m=+2042.595856585"
May 20 14:24:23.434067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:23.434040    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:23.434917 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:23.434891    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:24:24.437191 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:24.437153    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:24:29.442172 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:29.442143    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:24:29.442782 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:29.442750    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:24:39.443377 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:39.443289    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:24:49.443170 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:49.443128    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:24:59.443516 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:24:59.443473    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:25:09.443092 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:09.443049    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:25:19.442812 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:19.442772    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 20 14:25:20.458624 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:20.458597    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:25:20.461289 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:20.461266    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:25:20.463174 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:20.463154    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:25:20.465836 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:20.465818    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:25:29.444106 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:29.444069    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:25:36.042004 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.041968    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"]
May 20 14:25:36.042392 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.042355    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container"
May 20 14:25:36.042392 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.042370    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container"
May 20 14:25:36.042392 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.042389    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="storage-initializer"
May 20 14:25:36.042392 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.042395    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="storage-initializer"
May 20 14:25:36.042528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.042409    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kube-rbac-proxy"
May 20 14:25:36.042528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.042415    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kube-rbac-proxy"
May 20 14:25:36.042528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.042469    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kube-rbac-proxy"
May 20 14:25:36.042528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.042482    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="bdee6b7b-d8e1-49a5-ba10-3998b4dfe61e" containerName="kserve-container"
May 20 14:25:36.045873 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.045838    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.049794 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.049762    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9eb24d-predictor-serving-cert\""
May 20 14:25:36.049967 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.049878    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-9eb24d\""
May 20 14:25:36.050034 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.049985    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\""
May 20 14:25:36.050411 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.050390    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-9eb24d-dockercfg-s2nff\""
May 20 14:25:36.059784 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.059754    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"]
May 20 14:25:36.157456 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.157433    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-proxy-tls\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.157575 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.157472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjtt\" (UniqueName: \"kubernetes.io/projected/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kube-api-access-xpjtt\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.157575 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.157534    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.157664 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.157578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kserve-provision-location\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.157664 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.157627    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-cabundle-cert\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.258975 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.258945    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.259088 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.258983    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kserve-provision-location\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.259088 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.259012    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-cabundle-cert\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.259088 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.259043    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-proxy-tls\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.259244 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.259089    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjtt\" (UniqueName: \"kubernetes.io/projected/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kube-api-access-xpjtt\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.259302 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:36.259244    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-serving-cert: secret "isvc-secondary-9eb24d-predictor-serving-cert" not found
May 20 14:25:36.259374 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:36.259358    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-proxy-tls podName:57f2551b-c26d-4c3f-95ba-c2de8f9d991e nodeName:}" failed. No retries permitted until 2026-05-20 14:25:36.759334023 +0000 UTC m=+2116.896888252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-proxy-tls") pod "isvc-secondary-9eb24d-predictor-b57645d49-vgm5v" (UID: "57f2551b-c26d-4c3f-95ba-c2de8f9d991e") : secret "isvc-secondary-9eb24d-predictor-serving-cert" not found
May 20 14:25:36.259575 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.259552    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kserve-provision-location\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.259873 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.259828    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.259931 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.259879    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-cabundle-cert\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.268229 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.268202    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjtt\" (UniqueName: \"kubernetes.io/projected/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kube-api-access-xpjtt\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.762051 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.762015    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-proxy-tls\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.764595 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.764568    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-proxy-tls\") pod \"isvc-secondary-9eb24d-predictor-b57645d49-vgm5v\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") " pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:36.961245 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:36.961219    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:37.104750 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:37.104719    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"]
May 20 14:25:37.108277 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:25:37.108243    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f2551b_c26d_4c3f_95ba_c2de8f9d991e.slice/crio-34993142f0945da01e32d7fde150c8b73e1092c61fc07ca561b38210685409be WatchSource:0}: Error finding container 34993142f0945da01e32d7fde150c8b73e1092c61fc07ca561b38210685409be: Status 404 returned error can't find the container with id 34993142f0945da01e32d7fde150c8b73e1092c61fc07ca561b38210685409be
May 20 14:25:37.676740 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:37.676703    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v" event={"ID":"57f2551b-c26d-4c3f-95ba-c2de8f9d991e","Type":"ContainerStarted","Data":"6f2c1807919d4fb378b319bfa53b475ec52d4993115af5659b9cf27035e9b7e1"}
May 20 14:25:37.676740 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:37.676740    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v" event={"ID":"57f2551b-c26d-4c3f-95ba-c2de8f9d991e","Type":"ContainerStarted","Data":"34993142f0945da01e32d7fde150c8b73e1092c61fc07ca561b38210685409be"}
May 20 14:25:40.688056 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:40.688031    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_57f2551b-c26d-4c3f-95ba-c2de8f9d991e/storage-initializer/0.log"
May 20 14:25:40.688340 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:40.688070    2557 generic.go:358] "Generic (PLEG): container finished" podID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e" containerID="6f2c1807919d4fb378b319bfa53b475ec52d4993115af5659b9cf27035e9b7e1" exitCode=1
May 20 14:25:40.688340 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:40.688115    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v" event={"ID":"57f2551b-c26d-4c3f-95ba-c2de8f9d991e","Type":"ContainerDied","Data":"6f2c1807919d4fb378b319bfa53b475ec52d4993115af5659b9cf27035e9b7e1"}
May 20 14:25:41.693358 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:41.693331    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_57f2551b-c26d-4c3f-95ba-c2de8f9d991e/storage-initializer/0.log"
May 20 14:25:41.693724 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:41.693440    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v" event={"ID":"57f2551b-c26d-4c3f-95ba-c2de8f9d991e","Type":"ContainerStarted","Data":"b08967d5e45c4291a6ea9498874d117db8d1f8eb0f495c0292916237d750ccac"}
May 20 14:25:44.704753 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:44.704725    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_57f2551b-c26d-4c3f-95ba-c2de8f9d991e/storage-initializer/1.log"
May 20 14:25:44.705154 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:44.705138    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_57f2551b-c26d-4c3f-95ba-c2de8f9d991e/storage-initializer/0.log"
May 20 14:25:44.705201 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:44.705172    2557 generic.go:358] "Generic (PLEG): container finished" podID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e" containerID="b08967d5e45c4291a6ea9498874d117db8d1f8eb0f495c0292916237d750ccac" exitCode=1
May 20 14:25:44.705273 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:44.705250    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v" event={"ID":"57f2551b-c26d-4c3f-95ba-c2de8f9d991e","Type":"ContainerDied","Data":"b08967d5e45c4291a6ea9498874d117db8d1f8eb0f495c0292916237d750ccac"}
May 20 14:25:44.705344 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:44.705295    2557 scope.go:117] "RemoveContainer" containerID="6f2c1807919d4fb378b319bfa53b475ec52d4993115af5659b9cf27035e9b7e1"
May 20 14:25:44.705643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:44.705621    2557 scope.go:117] "RemoveContainer" containerID="6f2c1807919d4fb378b319bfa53b475ec52d4993115af5659b9cf27035e9b7e1"
May 20 14:25:44.715634 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:44.715610    2557 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_kserve-ci-e2e-test_57f2551b-c26d-4c3f-95ba-c2de8f9d991e_0 in pod sandbox 34993142f0945da01e32d7fde150c8b73e1092c61fc07ca561b38210685409be from index: no such id: '6f2c1807919d4fb378b319bfa53b475ec52d4993115af5659b9cf27035e9b7e1'" containerID="6f2c1807919d4fb378b319bfa53b475ec52d4993115af5659b9cf27035e9b7e1"
May 20 14:25:44.715718 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:44.715646    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2c1807919d4fb378b319bfa53b475ec52d4993115af5659b9cf27035e9b7e1"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_kserve-ci-e2e-test_57f2551b-c26d-4c3f-95ba-c2de8f9d991e_0 in pod sandbox 34993142f0945da01e32d7fde150c8b73e1092c61fc07ca561b38210685409be from index: no such id: '6f2c1807919d4fb378b319bfa53b475ec52d4993115af5659b9cf27035e9b7e1'"
May 20 14:25:44.715902 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:44.715878    2557 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_kserve-ci-e2e-test(57f2551b-c26d-4c3f-95ba-c2de8f9d991e)\"" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v" podUID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e"
May 20 14:25:45.709589 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:45.709563    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_57f2551b-c26d-4c3f-95ba-c2de8f9d991e/storage-initializer/1.log"
May 20 14:25:51.985553 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:51.985517    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"]
May 20 14:25:51.986034 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:51.985977    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container" containerID="cri-o://da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f" gracePeriod=30
May 20 14:25:51.986132 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:51.986012    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kube-rbac-proxy" containerID="cri-o://84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b" gracePeriod=30
May 20 14:25:52.056262 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.056235    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"]
May 20 14:25:52.180079 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.180052    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"]
May 20 14:25:52.183565 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.183544    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.186220 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.186196    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d20c43-predictor-serving-cert\""
May 20 14:25:52.186220 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.186216    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-d20c43-dockercfg-6l22k\""
May 20 14:25:52.186402 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.186219    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\""
May 20 14:25:52.186402 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.186201    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-d20c43\""
May 20 14:25:52.193256 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.193233    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"]
May 20 14:25:52.195801 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.195774    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_57f2551b-c26d-4c3f-95ba-c2de8f9d991e/storage-initializer/1.log"
May 20 14:25:52.195933 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.195909    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:52.276093 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276038    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpjtt\" (UniqueName: \"kubernetes.io/projected/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kube-api-access-xpjtt\") pod \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") "
May 20 14:25:52.276093 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276082    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\") pod \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") "
May 20 14:25:52.276235 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276119    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kserve-provision-location\") pod \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") "
May 20 14:25:52.276235 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276159    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-proxy-tls\") pod \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") "
May 20 14:25:52.276235 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276176    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-cabundle-cert\") pod \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\" (UID: \"57f2551b-c26d-4c3f-95ba-c2de8f9d991e\") "
May 20 14:25:52.276387 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276259    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.276387 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276315    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxmln\" (UniqueName: \"kubernetes.io/projected/f308d671-6927-4885-96c3-625f8a9ced68-kube-api-access-hxmln\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.276387 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276334    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f308d671-6927-4885-96c3-625f8a9ced68-kserve-provision-location\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.276546 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276425    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-cabundle-cert\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.276546 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276430    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57f2551b-c26d-4c3f-95ba-c2de8f9d991e" (UID: "57f2551b-c26d-4c3f-95ba-c2de8f9d991e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:52.276546 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276475    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f308d671-6927-4885-96c3-625f8a9ced68-proxy-tls\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.276546 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276518    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:52.276546 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276533    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-isvc-secondary-9eb24d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-9eb24d-kube-rbac-proxy-sar-config") pod "57f2551b-c26d-4c3f-95ba-c2de8f9d991e" (UID: "57f2551b-c26d-4c3f-95ba-c2de8f9d991e"). InnerVolumeSpecName "isvc-secondary-9eb24d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:25:52.276726 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.276589    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57f2551b-c26d-4c3f-95ba-c2de8f9d991e" (UID: "57f2551b-c26d-4c3f-95ba-c2de8f9d991e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:25:52.278143 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.278121    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kube-api-access-xpjtt" (OuterVolumeSpecName: "kube-api-access-xpjtt") pod "57f2551b-c26d-4c3f-95ba-c2de8f9d991e" (UID: "57f2551b-c26d-4c3f-95ba-c2de8f9d991e"). InnerVolumeSpecName "kube-api-access-xpjtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:25:52.278234 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.278207    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57f2551b-c26d-4c3f-95ba-c2de8f9d991e" (UID: "57f2551b-c26d-4c3f-95ba-c2de8f9d991e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:25:52.377596 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.377569    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmln\" (UniqueName: \"kubernetes.io/projected/f308d671-6927-4885-96c3-625f8a9ced68-kube-api-access-hxmln\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.377684 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.377597    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f308d671-6927-4885-96c3-625f8a9ced68-kserve-provision-location\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.377684 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.377628    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-cabundle-cert\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.377684 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.377645    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f308d671-6927-4885-96c3-625f8a9ced68-proxy-tls\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.377684 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.377681    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.377819 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.377711    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:52.377819 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.377721    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpjtt\" (UniqueName: \"kubernetes.io/projected/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-kube-api-access-xpjtt\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:52.377819 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.377736    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-isvc-secondary-9eb24d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:52.377819 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.377751    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f2551b-c26d-4c3f-95ba-c2de8f9d991e-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:52.377819 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:52.377812    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-serving-cert: secret "isvc-init-fail-d20c43-predictor-serving-cert" not found
May 20 14:25:52.378021 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:52.377903    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f308d671-6927-4885-96c3-625f8a9ced68-proxy-tls podName:f308d671-6927-4885-96c3-625f8a9ced68 nodeName:}" failed. No retries permitted until 2026-05-20 14:25:52.877884015 +0000 UTC m=+2133.015438224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f308d671-6927-4885-96c3-625f8a9ced68-proxy-tls") pod "isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn" (UID: "f308d671-6927-4885-96c3-625f8a9ced68") : secret "isvc-init-fail-d20c43-predictor-serving-cert" not found
May 20 14:25:52.378091 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.378066    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f308d671-6927-4885-96c3-625f8a9ced68-kserve-provision-location\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.378295 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.378280    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-cabundle-cert\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.378359 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.378344    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.386230 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.386212    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxmln\" (UniqueName: \"kubernetes.io/projected/f308d671-6927-4885-96c3-625f8a9ced68-kube-api-access-hxmln\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.738996 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.738968    2557 generic.go:358] "Generic (PLEG): container finished" podID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerID="84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b" exitCode=2
May 20 14:25:52.739125 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.739039    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" event={"ID":"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f","Type":"ContainerDied","Data":"84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b"}
May 20 14:25:52.740153 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.740139    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-9eb24d-predictor-b57645d49-vgm5v_57f2551b-c26d-4c3f-95ba-c2de8f9d991e/storage-initializer/1.log"
May 20 14:25:52.740246 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.740178    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v" event={"ID":"57f2551b-c26d-4c3f-95ba-c2de8f9d991e","Type":"ContainerDied","Data":"34993142f0945da01e32d7fde150c8b73e1092c61fc07ca561b38210685409be"}
May 20 14:25:52.740246 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.740206    2557 scope.go:117] "RemoveContainer" containerID="b08967d5e45c4291a6ea9498874d117db8d1f8eb0f495c0292916237d750ccac"
May 20 14:25:52.740323 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.740263    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"
May 20 14:25:52.774945 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.774922    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"]
May 20 14:25:52.780003 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.779985    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-9eb24d-predictor-b57645d49-vgm5v"]
May 20 14:25:52.880885 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.880847    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f308d671-6927-4885-96c3-625f8a9ced68-proxy-tls\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:52.883187 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:52.883167    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f308d671-6927-4885-96c3-625f8a9ced68-proxy-tls\") pod \"isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") " pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:53.108145 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:53.108121    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:25:53.236214 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:53.236193    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"]
May 20 14:25:53.238848 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:25:53.238809    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf308d671_6927_4885_96c3_625f8a9ced68.slice/crio-bc76c9036a0c6d94d042570d27b74ca4ea92deef3f416ad210e384f68733fd66 WatchSource:0}: Error finding container bc76c9036a0c6d94d042570d27b74ca4ea92deef3f416ad210e384f68733fd66: Status 404 returned error can't find the container with id bc76c9036a0c6d94d042570d27b74ca4ea92deef3f416ad210e384f68733fd66
May 20 14:25:53.746092 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:53.746056    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn" event={"ID":"f308d671-6927-4885-96c3-625f8a9ced68","Type":"ContainerStarted","Data":"5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f"}
May 20 14:25:53.746092 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:53.746097    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn" event={"ID":"f308d671-6927-4885-96c3-625f8a9ced68","Type":"ContainerStarted","Data":"bc76c9036a0c6d94d042570d27b74ca4ea92deef3f416ad210e384f68733fd66"}
May 20 14:25:54.405179 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:54.405146    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e" path="/var/lib/kubelet/pods/57f2551b-c26d-4c3f-95ba-c2de8f9d991e/volumes"
May 20 14:25:54.437900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:54.437839    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
May 20 14:25:55.729767 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.729739    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:25:55.755598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.755533    2557 generic.go:358] "Generic (PLEG): container finished" podID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerID="da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f" exitCode=0
May 20 14:25:55.755713 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.755619    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"
May 20 14:25:55.755713 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.755643    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" event={"ID":"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f","Type":"ContainerDied","Data":"da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f"}
May 20 14:25:55.755713 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.755685    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn" event={"ID":"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f","Type":"ContainerDied","Data":"1273cc173dcb32765935737b325708ed3f48e38736b2a4651008855fff5d1e0f"}
May 20 14:25:55.755713 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.755706    2557 scope.go:117] "RemoveContainer" containerID="84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b"
May 20 14:25:55.765892 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.765867    2557 scope.go:117] "RemoveContainer" containerID="da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f"
May 20 14:25:55.773968 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.773950    2557 scope.go:117] "RemoveContainer" containerID="a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88"
May 20 14:25:55.780522 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.780506    2557 scope.go:117] "RemoveContainer" containerID="84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b"
May 20 14:25:55.780783 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:55.780766    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b\": container with ID starting with 84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b not found: ID does not exist" containerID="84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b"
May 20 14:25:55.780824 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.780789    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b"} err="failed to get container status \"84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b\": rpc error: code = NotFound desc = could not find container \"84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b\": container with ID starting with 84fb359eace2567f3b176fd88acf1b0fa10eab91b2f5316444a6432aa481e68b not found: ID does not exist"
May 20 14:25:55.780824 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.780806    2557 scope.go:117] "RemoveContainer" containerID="da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f"
May 20 14:25:55.781068 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:55.781047    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f\": container with ID starting with da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f not found: ID does not exist" containerID="da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f"
May 20 14:25:55.781121 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.781074    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f"} err="failed to get container status \"da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f\": rpc error: code = NotFound desc = could not find container \"da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f\": container with ID starting with da0a5de8cd3b033902fb930e00c3e671ffa69fc8137d5f8aa3c0a35d570cc22f not found: ID does not exist"
May 20 14:25:55.781121 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.781090    2557 scope.go:117] "RemoveContainer" containerID="a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88"
May 20 14:25:55.781340 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:55.781324    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88\": container with ID starting with a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88 not found: ID does not exist" containerID="a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88"
May 20 14:25:55.781417 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.781345    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88"} err="failed to get container status \"a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88\": rpc error: code = NotFound desc = could not find container \"a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88\": container with ID starting with a1b61a76f9f0351f91fb751f3a8a0ac79b711a9cdc2473047a73e5e6d12a0c88 not found: ID does not exist"
May 20 14:25:55.802655 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.802636    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-cabundle-cert\") pod \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") "
May 20 14:25:55.802727 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.802685    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kserve-provision-location\") pod \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") "
May 20 14:25:55.802763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.802727    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-996vq\" (UniqueName: \"kubernetes.io/projected/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kube-api-access-996vq\") pod \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") "
May 20 14:25:55.802763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.802743    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-proxy-tls\") pod \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") "
May 20 14:25:55.802879 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.802769    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-isvc-primary-9eb24d-kube-rbac-proxy-sar-config\") pod \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\" (UID: \"6256f6d6-f1fd-42e5-847b-d8e85faf6c0f\") "
May 20 14:25:55.803109 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.803085    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" (UID: "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:55.803175 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.803014    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" (UID: "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:25:55.803231 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.803169    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-isvc-primary-9eb24d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-9eb24d-kube-rbac-proxy-sar-config") pod "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" (UID: "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f"). InnerVolumeSpecName "isvc-primary-9eb24d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:25:55.804665 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.804644    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kube-api-access-996vq" (OuterVolumeSpecName: "kube-api-access-996vq") pod "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" (UID: "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f"). InnerVolumeSpecName "kube-api-access-996vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:25:55.804736 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.804710    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" (UID: "6256f6d6-f1fd-42e5-847b-d8e85faf6c0f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:25:55.903765 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.903739    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:55.903765 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.903766    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-996vq\" (UniqueName: \"kubernetes.io/projected/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-kube-api-access-996vq\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:55.903918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.903778    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:55.903918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.903791    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-9eb24d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-isvc-primary-9eb24d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:55.903918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:55.903799    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:25:56.082942 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:56.082908    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"]
May 20 14:25:56.087519 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:56.087496    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9eb24d-predictor-86f8cd785d-wslfn"]
May 20 14:25:56.405689 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:56.405654    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" path="/var/lib/kubelet/pods/6256f6d6-f1fd-42e5-847b-d8e85faf6c0f/volumes"
May 20 14:25:56.761279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:56.761218    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn_f308d671-6927-4885-96c3-625f8a9ced68/storage-initializer/0.log"
May 20 14:25:56.761279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:56.761254    2557 generic.go:358] "Generic (PLEG): container finished" podID="f308d671-6927-4885-96c3-625f8a9ced68" containerID="5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f" exitCode=1
May 20 14:25:56.761631 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:56.761334    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn" event={"ID":"f308d671-6927-4885-96c3-625f8a9ced68","Type":"ContainerDied","Data":"5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f"}
May 20 14:25:57.178331 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.178300    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"]
May 20 14:25:57.318830 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.318799    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"]
May 20 14:25:57.319307 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319284    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kube-rbac-proxy"
May 20 14:25:57.319307 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319308    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kube-rbac-proxy"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319331    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319337    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319343    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e" containerName="storage-initializer"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319348    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e" containerName="storage-initializer"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319355    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e" containerName="storage-initializer"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319360    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e" containerName="storage-initializer"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319366    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="storage-initializer"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319371    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="storage-initializer"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319426    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e" containerName="storage-initializer"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319435    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kserve-container"
May 20 14:25:57.319486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319444    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="6256f6d6-f1fd-42e5-847b-d8e85faf6c0f" containerName="kube-rbac-proxy"
May 20 14:25:57.320023 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.319550    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="57f2551b-c26d-4c3f-95ba-c2de8f9d991e" containerName="storage-initializer"
May 20 14:25:57.322587 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.322568    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.325987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.325968    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:25:57.326127 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.326108    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
May 20 14:25:57.326231 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.326209    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
May 20 14:25:57.326362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.326252    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2mlds\""
May 20 14:25:57.335297 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.335272    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"]
May 20 14:25:57.415195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.415155    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.415195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.415187    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.415458 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.415236    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.415458 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.415282    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c9jj\" (UniqueName: \"kubernetes.io/projected/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kube-api-access-8c9jj\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.415458 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.415319    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718abd58-e978-4a3e-8fe6-01e0ffb95c08-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.516157 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.516083    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.516157 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.516123    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.516157 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.516144    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8c9jj\" (UniqueName: \"kubernetes.io/projected/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kube-api-access-8c9jj\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.516395 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.516188    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718abd58-e978-4a3e-8fe6-01e0ffb95c08-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.516395 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.516260    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.516395 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:57.516348    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
May 20 14:25:57.516551 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:25:57.516415    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/718abd58-e978-4a3e-8fe6-01e0ffb95c08-proxy-tls podName:718abd58-e978-4a3e-8fe6-01e0ffb95c08 nodeName:}" failed. No retries permitted until 2026-05-20 14:25:58.016395451 +0000 UTC m=+2138.153949662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/718abd58-e978-4a3e-8fe6-01e0ffb95c08-proxy-tls") pod "isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" (UID: "718abd58-e978-4a3e-8fe6-01e0ffb95c08") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
May 20 14:25:57.516659 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.516641    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.516913 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.516890    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.517003 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.516949    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.525262 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.525241    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c9jj\" (UniqueName: \"kubernetes.io/projected/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kube-api-access-8c9jj\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:57.772083 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.772022    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn_f308d671-6927-4885-96c3-625f8a9ced68/storage-initializer/0.log"
May 20 14:25:57.772444 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.772103    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn" event={"ID":"f308d671-6927-4885-96c3-625f8a9ced68","Type":"ContainerStarted","Data":"9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674"}
May 20 14:25:57.772444 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:57.772239    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn" podUID="f308d671-6927-4885-96c3-625f8a9ced68" containerName="storage-initializer" containerID="cri-o://9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674" gracePeriod=30
May 20 14:25:58.021098 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:58.021058    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718abd58-e978-4a3e-8fe6-01e0ffb95c08-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:58.023673 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:58.023619    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718abd58-e978-4a3e-8fe6-01e0ffb95c08-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:58.232469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:58.232438    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:25:58.359416 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:58.359389    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"]
May 20 14:25:58.360929 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:25:58.360895    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718abd58_e978_4a3e_8fe6_01e0ffb95c08.slice/crio-3e4092d7cd8fe468040f4d2e5ed49113ee25f8fa4e9670bed1f8ce948e6b5f26 WatchSource:0}: Error finding container 3e4092d7cd8fe468040f4d2e5ed49113ee25f8fa4e9670bed1f8ce948e6b5f26: Status 404 returned error can't find the container with id 3e4092d7cd8fe468040f4d2e5ed49113ee25f8fa4e9670bed1f8ce948e6b5f26
May 20 14:25:58.777174 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:58.777143    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" event={"ID":"718abd58-e978-4a3e-8fe6-01e0ffb95c08","Type":"ContainerStarted","Data":"8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b"}
May 20 14:25:58.777174 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:25:58.777177    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" event={"ID":"718abd58-e978-4a3e-8fe6-01e0ffb95c08","Type":"ContainerStarted","Data":"3e4092d7cd8fe468040f4d2e5ed49113ee25f8fa4e9670bed1f8ce948e6b5f26"}
May 20 14:26:02.518608 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.518586    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn_f308d671-6927-4885-96c3-625f8a9ced68/storage-initializer/1.log"
May 20 14:26:02.519002 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.518959    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn_f308d671-6927-4885-96c3-625f8a9ced68/storage-initializer/0.log"
May 20 14:26:02.519069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.519052    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:26:02.659099 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.659038    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f308d671-6927-4885-96c3-625f8a9ced68-kserve-provision-location\") pod \"f308d671-6927-4885-96c3-625f8a9ced68\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") "
May 20 14:26:02.659099 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.659095    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\") pod \"f308d671-6927-4885-96c3-625f8a9ced68\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") "
May 20 14:26:02.659285 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.659128    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f308d671-6927-4885-96c3-625f8a9ced68-proxy-tls\") pod \"f308d671-6927-4885-96c3-625f8a9ced68\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") "
May 20 14:26:02.659285 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.659145    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxmln\" (UniqueName: \"kubernetes.io/projected/f308d671-6927-4885-96c3-625f8a9ced68-kube-api-access-hxmln\") pod \"f308d671-6927-4885-96c3-625f8a9ced68\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") "
May 20 14:26:02.659285 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.659160    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-cabundle-cert\") pod \"f308d671-6927-4885-96c3-625f8a9ced68\" (UID: \"f308d671-6927-4885-96c3-625f8a9ced68\") "
May 20 14:26:02.659442 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.659363    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f308d671-6927-4885-96c3-625f8a9ced68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f308d671-6927-4885-96c3-625f8a9ced68" (UID: "f308d671-6927-4885-96c3-625f8a9ced68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:26:02.659548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.659524    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-isvc-init-fail-d20c43-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-d20c43-kube-rbac-proxy-sar-config") pod "f308d671-6927-4885-96c3-625f8a9ced68" (UID: "f308d671-6927-4885-96c3-625f8a9ced68"). InnerVolumeSpecName "isvc-init-fail-d20c43-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:26:02.659590 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.659547    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f308d671-6927-4885-96c3-625f8a9ced68" (UID: "f308d671-6927-4885-96c3-625f8a9ced68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:26:02.661225 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.661203    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f308d671-6927-4885-96c3-625f8a9ced68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f308d671-6927-4885-96c3-625f8a9ced68" (UID: "f308d671-6927-4885-96c3-625f8a9ced68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:26:02.661324 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.661287    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f308d671-6927-4885-96c3-625f8a9ced68-kube-api-access-hxmln" (OuterVolumeSpecName: "kube-api-access-hxmln") pod "f308d671-6927-4885-96c3-625f8a9ced68" (UID: "f308d671-6927-4885-96c3-625f8a9ced68"). InnerVolumeSpecName "kube-api-access-hxmln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:26:02.759816 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.759794    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f308d671-6927-4885-96c3-625f8a9ced68-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:26:02.759816 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.759814    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxmln\" (UniqueName: \"kubernetes.io/projected/f308d671-6927-4885-96c3-625f8a9ced68-kube-api-access-hxmln\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:26:02.759956 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.759824    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:26:02.759956 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.759834    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f308d671-6927-4885-96c3-625f8a9ced68-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:26:02.759956 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.759843    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f308d671-6927-4885-96c3-625f8a9ced68-isvc-init-fail-d20c43-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:26:02.797796 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.797776    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn_f308d671-6927-4885-96c3-625f8a9ced68/storage-initializer/1.log"
May 20 14:26:02.798147 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.798133    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn_f308d671-6927-4885-96c3-625f8a9ced68/storage-initializer/0.log"
May 20 14:26:02.798207 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.798166    2557 generic.go:358] "Generic (PLEG): container finished" podID="f308d671-6927-4885-96c3-625f8a9ced68" containerID="9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674" exitCode=1
May 20 14:26:02.798259 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.798201    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn" event={"ID":"f308d671-6927-4885-96c3-625f8a9ced68","Type":"ContainerDied","Data":"9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674"}
May 20 14:26:02.798259 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.798244    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"
May 20 14:26:02.798366 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.798263    2557 scope.go:117] "RemoveContainer" containerID="9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674"
May 20 14:26:02.798366 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.798249    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn" event={"ID":"f308d671-6927-4885-96c3-625f8a9ced68","Type":"ContainerDied","Data":"bc76c9036a0c6d94d042570d27b74ca4ea92deef3f416ad210e384f68733fd66"}
May 20 14:26:02.799687 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.799659    2557 generic.go:358] "Generic (PLEG): container finished" podID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerID="8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b" exitCode=0
May 20 14:26:02.799794 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.799701    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" event={"ID":"718abd58-e978-4a3e-8fe6-01e0ffb95c08","Type":"ContainerDied","Data":"8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b"}
May 20 14:26:02.806814 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.806784    2557 scope.go:117] "RemoveContainer" containerID="5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f"
May 20 14:26:02.814740 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.814722    2557 scope.go:117] "RemoveContainer" containerID="9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674"
May 20 14:26:02.814994 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:26:02.814979    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674\": container with ID starting with 9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674 not found: ID does not exist" containerID="9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674"
May 20 14:26:02.815048 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.814999    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674"} err="failed to get container status \"9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674\": rpc error: code = NotFound desc = could not find container \"9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674\": container with ID starting with 9b9e0ae8afc3e734b8b98b81957b646c960b5e69f4f20ba7298deada7829a674 not found: ID does not exist"
May 20 14:26:02.815048 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.815015    2557 scope.go:117] "RemoveContainer" containerID="5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f"
May 20 14:26:02.815262 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:26:02.815244    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f\": container with ID starting with 5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f not found: ID does not exist" containerID="5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f"
May 20 14:26:02.815310 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.815267    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f"} err="failed to get container status \"5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f\": rpc error: code = NotFound desc = could not find container \"5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f\": container with ID starting with 5edcb8249bca3cc5e67c492580bc348fddf6aadb31ba3bf51845616b0896fc5f not found: ID does not exist"
May 20 14:26:02.861632 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.861611    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"]
May 20 14:26:02.866837 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:02.866817    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d20c43-predictor-77cb89f4f7-p8bfn"]
May 20 14:26:04.407408 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:04.407074    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f308d671-6927-4885-96c3-625f8a9ced68" path="/var/lib/kubelet/pods/f308d671-6927-4885-96c3-625f8a9ced68/volumes"
May 20 14:26:24.897838 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:24.897810    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" event={"ID":"718abd58-e978-4a3e-8fe6-01e0ffb95c08","Type":"ContainerStarted","Data":"bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7"}
May 20 14:26:24.906778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:24.897866    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" event={"ID":"718abd58-e978-4a3e-8fe6-01e0ffb95c08","Type":"ContainerStarted","Data":"a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da"}
May 20 14:26:24.906778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:24.898108    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:26:24.922952 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:24.922912    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podStartSLOduration=6.553228282 podStartE2EDuration="27.922900334s" podCreationTimestamp="2026-05-20 14:25:57 +0000 UTC" firstStartedPulling="2026-05-20 14:26:02.801000759 +0000 UTC m=+2142.938554955" lastFinishedPulling="2026-05-20 14:26:24.170672795 +0000 UTC m=+2164.308227007" observedRunningTime="2026-05-20 14:26:24.920928637 +0000 UTC m=+2165.058482854" watchObservedRunningTime="2026-05-20 14:26:24.922900334 +0000 UTC m=+2165.060454552"
May 20 14:26:25.901483 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:25.901459    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:26:25.902578 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:25.902552    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 20 14:26:26.904942 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:26.904902    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 20 14:26:31.909415 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:31.909389    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:26:31.910047 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:31.910015    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 20 14:26:41.910352 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:41.910307    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 20 14:26:51.910936 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:26:51.910889    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 20 14:27:01.910366 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:01.910321    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 20 14:27:11.910275 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:11.910233    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 20 14:27:21.910322 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:21.910277    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 20 14:27:31.910638 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:31.910544    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 20 14:27:41.911050 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:41.911010    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:27:47.557286 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.557255    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"]
May 20 14:27:47.557746 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.557545    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container" containerID="cri-o://a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da" gracePeriod=30
May 20 14:27:47.557746 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.557586    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kube-rbac-proxy" containerID="cri-o://bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7" gracePeriod=30
May 20 14:27:47.736464 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.736429    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"]
May 20 14:27:47.736827 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.736809    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f308d671-6927-4885-96c3-625f8a9ced68" containerName="storage-initializer"
May 20 14:27:47.736946 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.736828    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f308d671-6927-4885-96c3-625f8a9ced68" containerName="storage-initializer"
May 20 14:27:47.736946 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.736873    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f308d671-6927-4885-96c3-625f8a9ced68" containerName="storage-initializer"
May 20 14:27:47.736946 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.736882    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f308d671-6927-4885-96c3-625f8a9ced68" containerName="storage-initializer"
May 20 14:27:47.737109 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.736968    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f308d671-6927-4885-96c3-625f8a9ced68" containerName="storage-initializer"
May 20 14:27:47.737109 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.736982    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="f308d671-6927-4885-96c3-625f8a9ced68" containerName="storage-initializer"
May 20 14:27:47.740101 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.740083    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.749114 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.749094    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
May 20 14:27:47.749312 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.749297    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
May 20 14:27:47.770918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.770891    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"]
May 20 14:27:47.844796 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.844759    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.844913 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.844814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.844913 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.844890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxzr\" (UniqueName: \"kubernetes.io/projected/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kube-api-access-szxzr\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.845037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.844947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.845037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.844999    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.945981 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.945959    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.946078 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.945989    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.946078 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.946044    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.946078 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.946069    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.946243 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.946095    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-szxzr\" (UniqueName: \"kubernetes.io/projected/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kube-api-access-szxzr\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.946387 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.946362    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.946670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.946638    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.946761 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.946680    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.948401 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.948384    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:47.956216 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:47.956195    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxzr\" (UniqueName: \"kubernetes.io/projected/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kube-api-access-szxzr\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-747x9\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:48.049816 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:48.049794    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:48.162981 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:48.162953    2557 generic.go:358] "Generic (PLEG): container finished" podID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerID="bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7" exitCode=2
May 20 14:27:48.163100 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:48.163008    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" event={"ID":"718abd58-e978-4a3e-8fe6-01e0ffb95c08","Type":"ContainerDied","Data":"bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7"}
May 20 14:27:48.176369 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:48.176347    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"]
May 20 14:27:48.178845 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:27:48.178822    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55a76a3_e0a7_46ef_ae9d_e840574c85fb.slice/crio-c2c139756f41a0b33676b30a55fb54b6655e7ddb5c03baa671ab41f8db389f12 WatchSource:0}: Error finding container c2c139756f41a0b33676b30a55fb54b6655e7ddb5c03baa671ab41f8db389f12: Status 404 returned error can't find the container with id c2c139756f41a0b33676b30a55fb54b6655e7ddb5c03baa671ab41f8db389f12
May 20 14:27:49.167872 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:49.167832    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" event={"ID":"c55a76a3-e0a7-46ef-ae9d-e840574c85fb","Type":"ContainerStarted","Data":"14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457"}
May 20 14:27:49.167872 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:49.167878    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" event={"ID":"c55a76a3-e0a7-46ef-ae9d-e840574c85fb","Type":"ContainerStarted","Data":"c2c139756f41a0b33676b30a55fb54b6655e7ddb5c03baa671ab41f8db389f12"}
May 20 14:27:51.695699 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.695677    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:27:51.874961 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.874934    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") "
May 20 14:27:51.875087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.874969    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kserve-provision-location\") pod \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") "
May 20 14:27:51.875087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.874998    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718abd58-e978-4a3e-8fe6-01e0ffb95c08-proxy-tls\") pod \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") "
May 20 14:27:51.875087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.875040    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-cabundle-cert\") pod \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") "
May 20 14:27:51.875087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.875063    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c9jj\" (UniqueName: \"kubernetes.io/projected/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kube-api-access-8c9jj\") pod \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\" (UID: \"718abd58-e978-4a3e-8fe6-01e0ffb95c08\") "
May 20 14:27:51.875427 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.875388    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "718abd58-e978-4a3e-8fe6-01e0ffb95c08" (UID: "718abd58-e978-4a3e-8fe6-01e0ffb95c08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:27:51.875427 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.875396    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "718abd58-e978-4a3e-8fe6-01e0ffb95c08" (UID: "718abd58-e978-4a3e-8fe6-01e0ffb95c08"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:27:51.875570 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.875432    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "718abd58-e978-4a3e-8fe6-01e0ffb95c08" (UID: "718abd58-e978-4a3e-8fe6-01e0ffb95c08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:27:51.877286 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.877258    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kube-api-access-8c9jj" (OuterVolumeSpecName: "kube-api-access-8c9jj") pod "718abd58-e978-4a3e-8fe6-01e0ffb95c08" (UID: "718abd58-e978-4a3e-8fe6-01e0ffb95c08"). InnerVolumeSpecName "kube-api-access-8c9jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:27:51.877399 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.877264    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/718abd58-e978-4a3e-8fe6-01e0ffb95c08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "718abd58-e978-4a3e-8fe6-01e0ffb95c08" (UID: "718abd58-e978-4a3e-8fe6-01e0ffb95c08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:27:51.975907 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.975881    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:27:51.975907 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.975905    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c9jj\" (UniqueName: \"kubernetes.io/projected/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kube-api-access-8c9jj\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:27:51.976038 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.975917    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/718abd58-e978-4a3e-8fe6-01e0ffb95c08-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:27:51.976038 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.975927    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/718abd58-e978-4a3e-8fe6-01e0ffb95c08-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:27:51.976038 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:51.975936    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/718abd58-e978-4a3e-8fe6-01e0ffb95c08-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:27:52.178880 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.178803    2557 generic.go:358] "Generic (PLEG): container finished" podID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerID="a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da" exitCode=0
May 20 14:27:52.178989 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.178895    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" event={"ID":"718abd58-e978-4a3e-8fe6-01e0ffb95c08","Type":"ContainerDied","Data":"a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da"}
May 20 14:27:52.178989 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.178916    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"
May 20 14:27:52.178989 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.178941    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl" event={"ID":"718abd58-e978-4a3e-8fe6-01e0ffb95c08","Type":"ContainerDied","Data":"3e4092d7cd8fe468040f4d2e5ed49113ee25f8fa4e9670bed1f8ce948e6b5f26"}
May 20 14:27:52.178989 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.178963    2557 scope.go:117] "RemoveContainer" containerID="bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7"
May 20 14:27:52.180410 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.180382    2557 generic.go:358] "Generic (PLEG): container finished" podID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerID="14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457" exitCode=0
May 20 14:27:52.180538 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.180439    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" event={"ID":"c55a76a3-e0a7-46ef-ae9d-e840574c85fb","Type":"ContainerDied","Data":"14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457"}
May 20 14:27:52.187515 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.187459    2557 scope.go:117] "RemoveContainer" containerID="a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da"
May 20 14:27:52.194977 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.194958    2557 scope.go:117] "RemoveContainer" containerID="8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b"
May 20 14:27:52.201866 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.201839    2557 scope.go:117] "RemoveContainer" containerID="bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7"
May 20 14:27:52.202146 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:27:52.202122    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7\": container with ID starting with bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7 not found: ID does not exist" containerID="bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7"
May 20 14:27:52.202232 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.202160    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7"} err="failed to get container status \"bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7\": rpc error: code = NotFound desc = could not find container \"bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7\": container with ID starting with bd8a99757bc5a3ffa71d1bef15a959ec1d5e30e855de8d784c91df9f1128f9c7 not found: ID does not exist"
May 20 14:27:52.202232 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.202184    2557 scope.go:117] "RemoveContainer" containerID="a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da"
May 20 14:27:52.202446 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:27:52.202426    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da\": container with ID starting with a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da not found: ID does not exist" containerID="a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da"
May 20 14:27:52.202495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.202453    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da"} err="failed to get container status \"a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da\": rpc error: code = NotFound desc = could not find container \"a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da\": container with ID starting with a60eee543d1de98e404e6a70faf1b6fe6a89fe6ff57d40cde3d6338b140f61da not found: ID does not exist"
May 20 14:27:52.202495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.202474    2557 scope.go:117] "RemoveContainer" containerID="8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b"
May 20 14:27:52.202780 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:27:52.202754    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b\": container with ID starting with 8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b not found: ID does not exist" containerID="8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b"
May 20 14:27:52.202780 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.202782    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b"} err="failed to get container status \"8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b\": rpc error: code = NotFound desc = could not find container \"8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b\": container with ID starting with 8381a82e76569f7ae0b89e62d13d6b4a186940f909478dbda3c6ae7dfc4c912b not found: ID does not exist"
May 20 14:27:52.219095 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.219074    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"]
May 20 14:27:52.222965 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.222945    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-mlpnl"]
May 20 14:27:52.404829 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:52.404798    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" path="/var/lib/kubelet/pods/718abd58-e978-4a3e-8fe6-01e0ffb95c08/volumes"
May 20 14:27:53.187362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:53.187327    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" event={"ID":"c55a76a3-e0a7-46ef-ae9d-e840574c85fb","Type":"ContainerStarted","Data":"0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c"}
May 20 14:27:53.187362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:53.187362    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" event={"ID":"c55a76a3-e0a7-46ef-ae9d-e840574c85fb","Type":"ContainerStarted","Data":"4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7"}
May 20 14:27:53.187791 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:53.187703    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:53.187871 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:53.187838    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:27:53.188915 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:53.188884    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:27:53.209657 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:53.209611    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podStartSLOduration=6.209600048 podStartE2EDuration="6.209600048s" podCreationTimestamp="2026-05-20 14:27:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:27:53.208462935 +0000 UTC m=+2253.346017154" watchObservedRunningTime="2026-05-20 14:27:53.209600048 +0000 UTC m=+2253.347154265"
May 20 14:27:54.191622 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:54.191580    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:27:55.194823 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:27:55.194781    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:28:00.199804 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:28:00.199776    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:28:00.200451 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:28:00.200420    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:28:10.201202 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:28:10.201163    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:28:20.200350 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:28:20.200309    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:28:30.201361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:28:30.201318    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:28:40.200947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:28:40.200906    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:28:50.200726 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:28:50.200679    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:29:00.201040 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:00.201002    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:29:07.736770 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.736696    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"]
May 20 14:29:07.737252 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.737004    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" containerID="cri-o://4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7" gracePeriod=30
May 20 14:29:07.737252 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.737056    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kube-rbac-proxy" containerID="cri-o://0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c" gracePeriod=30
May 20 14:29:07.824097 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.824060    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"]
May 20 14:29:07.824438 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.824424    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kube-rbac-proxy"
May 20 14:29:07.824482 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.824439    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kube-rbac-proxy"
May 20 14:29:07.824482 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.824465    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container"
May 20 14:29:07.824482 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.824471    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container"
May 20 14:29:07.824583 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.824487    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="storage-initializer"
May 20 14:29:07.824583 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.824493    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="storage-initializer"
May 20 14:29:07.824583 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.824541    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kserve-container"
May 20 14:29:07.824583 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.824549    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="718abd58-e978-4a3e-8fe6-01e0ffb95c08" containerName="kube-rbac-proxy"
May 20 14:29:07.827729 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.827714    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:07.832523 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.832501    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
May 20 14:29:07.832625 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.832608    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
May 20 14:29:07.838590 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.838569    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"]
May 20 14:29:07.936963 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.936929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:07.937102 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.937012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd356406-964f-49ee-8ae0-115658c0170f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:07.937102 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.937047    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:07.937102 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.937073    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/bd356406-964f-49ee-8ae0-115658c0170f-kube-api-access-dw84v\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:07.937213 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:07.937119    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd356406-964f-49ee-8ae0-115658c0170f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.037912 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.037815    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd356406-964f-49ee-8ae0-115658c0170f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.037912 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.037846    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.038093 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.037966    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/bd356406-964f-49ee-8ae0-115658c0170f-kube-api-access-dw84v\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.038093 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.038020    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd356406-964f-49ee-8ae0-115658c0170f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.038093 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.038082    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.038422 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.038398    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd356406-964f-49ee-8ae0-115658c0170f-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.038629 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.038611    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.038723 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.038706    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.040362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.040342    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd356406-964f-49ee-8ae0-115658c0170f-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.048152 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.048133    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/bd356406-964f-49ee-8ae0-115658c0170f-kube-api-access-dw84v\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.137814 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.137775    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:08.263944 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.263918    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"]
May 20 14:29:08.265893 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:29:08.265867    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd356406_964f_49ee_8ae0_115658c0170f.slice/crio-d031fe80dab4435ffd5c35e3724fc2b067c4b2d15da732f87ca29d5e028a28c3 WatchSource:0}: Error finding container d031fe80dab4435ffd5c35e3724fc2b067c4b2d15da732f87ca29d5e028a28c3: Status 404 returned error can't find the container with id d031fe80dab4435ffd5c35e3724fc2b067c4b2d15da732f87ca29d5e028a28c3
May 20 14:29:08.430548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.430508    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" event={"ID":"bd356406-964f-49ee-8ae0-115658c0170f","Type":"ContainerStarted","Data":"8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764"}
May 20 14:29:08.430548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.430550    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" event={"ID":"bd356406-964f-49ee-8ae0-115658c0170f","Type":"ContainerStarted","Data":"d031fe80dab4435ffd5c35e3724fc2b067c4b2d15da732f87ca29d5e028a28c3"}
May 20 14:29:08.432488 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.432460    2557 generic.go:358] "Generic (PLEG): container finished" podID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerID="0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c" exitCode=2
May 20 14:29:08.432607 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:08.432504    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" event={"ID":"c55a76a3-e0a7-46ef-ae9d-e840574c85fb","Type":"ContainerDied","Data":"0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c"}
May 20 14:29:10.195643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:10.195591    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
May 20 14:29:10.200970 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:10.200941    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 20 14:29:12.114362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.114343    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:29:12.271388 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.271326    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") "
May 20 14:29:12.271494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.271427    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-cabundle-cert\") pod \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") "
May 20 14:29:12.271494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.271453    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szxzr\" (UniqueName: \"kubernetes.io/projected/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kube-api-access-szxzr\") pod \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") "
May 20 14:29:12.271494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.271483    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-proxy-tls\") pod \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") "
May 20 14:29:12.271673 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.271513    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kserve-provision-location\") pod \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\" (UID: \"c55a76a3-e0a7-46ef-ae9d-e840574c85fb\") "
May 20 14:29:12.271749 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.271637    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "c55a76a3-e0a7-46ef-ae9d-e840574c85fb" (UID: "c55a76a3-e0a7-46ef-ae9d-e840574c85fb"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:29:12.271823 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.271796    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c55a76a3-e0a7-46ef-ae9d-e840574c85fb" (UID: "c55a76a3-e0a7-46ef-ae9d-e840574c85fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:29:12.271946 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.271847    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c55a76a3-e0a7-46ef-ae9d-e840574c85fb" (UID: "c55a76a3-e0a7-46ef-ae9d-e840574c85fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:29:12.273574 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.273551    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kube-api-access-szxzr" (OuterVolumeSpecName: "kube-api-access-szxzr") pod "c55a76a3-e0a7-46ef-ae9d-e840574c85fb" (UID: "c55a76a3-e0a7-46ef-ae9d-e840574c85fb"). InnerVolumeSpecName "kube-api-access-szxzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:29:12.273647 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.273562    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c55a76a3-e0a7-46ef-ae9d-e840574c85fb" (UID: "c55a76a3-e0a7-46ef-ae9d-e840574c85fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:29:12.372742 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.372716    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:29:12.372742 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.372739    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szxzr\" (UniqueName: \"kubernetes.io/projected/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kube-api-access-szxzr\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:29:12.372899 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.372749    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:29:12.372899 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.372758    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:29:12.372899 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.372768    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c55a76a3-e0a7-46ef-ae9d-e840574c85fb-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:29:12.447653 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.447623    2557 generic.go:358] "Generic (PLEG): container finished" podID="bd356406-964f-49ee-8ae0-115658c0170f" containerID="8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764" exitCode=0
May 20 14:29:12.447779 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.447702    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" event={"ID":"bd356406-964f-49ee-8ae0-115658c0170f","Type":"ContainerDied","Data":"8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764"}
May 20 14:29:12.449534 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.449511    2557 generic.go:358] "Generic (PLEG): container finished" podID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerID="4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7" exitCode=0
May 20 14:29:12.449618 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.449553    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" event={"ID":"c55a76a3-e0a7-46ef-ae9d-e840574c85fb","Type":"ContainerDied","Data":"4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7"}
May 20 14:29:12.449618 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.449576    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9" event={"ID":"c55a76a3-e0a7-46ef-ae9d-e840574c85fb","Type":"ContainerDied","Data":"c2c139756f41a0b33676b30a55fb54b6655e7ddb5c03baa671ab41f8db389f12"}
May 20 14:29:12.449618 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.449578    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"
May 20 14:29:12.449741 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.449647    2557 scope.go:117] "RemoveContainer" containerID="0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c"
May 20 14:29:12.457679 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.457646    2557 scope.go:117] "RemoveContainer" containerID="4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7"
May 20 14:29:12.464846 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.464831    2557 scope.go:117] "RemoveContainer" containerID="14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457"
May 20 14:29:12.472938 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.472900    2557 scope.go:117] "RemoveContainer" containerID="0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c"
May 20 14:29:12.473231 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:29:12.473209    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c\": container with ID starting with 0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c not found: ID does not exist" containerID="0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c"
May 20 14:29:12.473341 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.473243    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c"} err="failed to get container status \"0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c\": rpc error: code = NotFound desc = could not find container \"0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c\": container with ID starting with 0dbbc01ed1fa1b2490dc21af91b75d94602ed5de686d1e30f4eb8bdcbecdb57c not found: ID does not exist"
May 20 14:29:12.473341 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.473266    2557 scope.go:117] "RemoveContainer" containerID="4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7"
May 20 14:29:12.473539 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:29:12.473514    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7\": container with ID starting with 4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7 not found: ID does not exist" containerID="4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7"
May 20 14:29:12.473582 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.473546    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7"} err="failed to get container status \"4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7\": rpc error: code = NotFound desc = could not find container \"4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7\": container with ID starting with 4cffb09180e300e440c678d9c8fadbd73fe9b5a72d1e16ecc347af472af916f7 not found: ID does not exist"
May 20 14:29:12.473582 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.473562    2557 scope.go:117] "RemoveContainer" containerID="14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457"
May 20 14:29:12.473914 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:29:12.473833    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457\": container with ID starting with 14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457 not found: ID does not exist" containerID="14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457"
May 20 14:29:12.473914 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.473883    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457"} err="failed to get container status \"14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457\": rpc error: code = NotFound desc = could not find container \"14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457\": container with ID starting with 14170f947135f895523bec94e31926356c8e8e24ae6385307ac5f21c5cd70457 not found: ID does not exist"
May 20 14:29:12.482537 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.482515    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"]
May 20 14:29:12.487805 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:12.487784    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-747x9"]
May 20 14:29:13.453989 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:13.453953    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" event={"ID":"bd356406-964f-49ee-8ae0-115658c0170f","Type":"ContainerStarted","Data":"b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927"}
May 20 14:29:13.454495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:13.453996    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" event={"ID":"bd356406-964f-49ee-8ae0-115658c0170f","Type":"ContainerStarted","Data":"ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22"}
May 20 14:29:13.454495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:13.454242    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:13.475420 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:13.475376    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podStartSLOduration=6.475362856 podStartE2EDuration="6.475362856s" podCreationTimestamp="2026-05-20 14:29:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:29:13.473080307 +0000 UTC m=+2333.610634527" watchObservedRunningTime="2026-05-20 14:29:13.475362856 +0000 UTC m=+2333.612917075"
May 20 14:29:14.405256 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:14.405217    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" path="/var/lib/kubelet/pods/c55a76a3-e0a7-46ef-ae9d-e840574c85fb/volumes"
May 20 14:29:14.457837 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:14.457809    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:14.458980 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:14.458954    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:29:15.461969 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:15.461912    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:29:20.466448 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:20.466419    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:29:20.467053 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:20.467015    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:29:30.467774 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:30.467729    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:29:40.467010 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:40.466967    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:29:50.467385 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:29:50.467341    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:30:00.467922 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:00.467874    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:30:10.467514 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:10.467478    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:30:20.467928 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:20.467892    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:30:20.491349 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:20.491325    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:30:20.495067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:20.495042    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:30:20.496343 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:20.496321    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:30:20.499614 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:20.499592    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:30:30.467919 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:30.467886    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:30:38.010573 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.010539    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"]
May 20 14:30:38.011161 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.010953    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" containerID="cri-o://ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22" gracePeriod=30
May 20 14:30:38.011161 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.010965    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kube-rbac-proxy" containerID="cri-o://b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927" gracePeriod=30
May 20 14:30:38.268575 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.268506    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"]
May 20 14:30:38.268894 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.268882    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container"
May 20 14:30:38.268947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.268896    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container"
May 20 14:30:38.268947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.268906    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="storage-initializer"
May 20 14:30:38.268947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.268912    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="storage-initializer"
May 20 14:30:38.268947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.268938    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kube-rbac-proxy"
May 20 14:30:38.268947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.268943    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kube-rbac-proxy"
May 20 14:30:38.269120 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.268993    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kserve-container"
May 20 14:30:38.269120 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.269008    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="c55a76a3-e0a7-46ef-ae9d-e840574c85fb" containerName="kube-rbac-proxy"
May 20 14:30:38.272072 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.272056    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.275316 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.275298    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
May 20 14:30:38.275663 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.275648    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
May 20 14:30:38.297471 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.297447    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"]
May 20 14:30:38.410916 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.410890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.411061 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.410925    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96de3b-d9a0-430d-8ce4-20063e730550-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.411061 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.410961    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.411061 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.411011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklcf\" (UniqueName: \"kubernetes.io/projected/6e96de3b-d9a0-430d-8ce4-20063e730550-kube-api-access-jklcf\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.411061 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.411043    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96de3b-d9a0-430d-8ce4-20063e730550-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.512316 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.512283    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.512503 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.512321    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96de3b-d9a0-430d-8ce4-20063e730550-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.512503 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.512354    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.512503 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.512377    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jklcf\" (UniqueName: \"kubernetes.io/projected/6e96de3b-d9a0-430d-8ce4-20063e730550-kube-api-access-jklcf\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.512503 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.512394    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96de3b-d9a0-430d-8ce4-20063e730550-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.512683 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:30:38.512530    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
May 20 14:30:38.512683 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:30:38.512579    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e96de3b-d9a0-430d-8ce4-20063e730550-proxy-tls podName:6e96de3b-d9a0-430d-8ce4-20063e730550 nodeName:}" failed. No retries permitted until 2026-05-20 14:30:39.012564569 +0000 UTC m=+2419.150118766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e96de3b-d9a0-430d-8ce4-20063e730550-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" (UID: "6e96de3b-d9a0-430d-8ce4-20063e730550") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
May 20 14:30:38.512766 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.512749    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96de3b-d9a0-430d-8ce4-20063e730550-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.512995 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.512978    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.513052 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.513033    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.531421 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.531359    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklcf\" (UniqueName: \"kubernetes.io/projected/6e96de3b-d9a0-430d-8ce4-20063e730550-kube-api-access-jklcf\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:38.720583 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.720547    2557 generic.go:358] "Generic (PLEG): container finished" podID="bd356406-964f-49ee-8ae0-115658c0170f" containerID="b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927" exitCode=2
May 20 14:30:38.720583 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:38.720584    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" event={"ID":"bd356406-964f-49ee-8ae0-115658c0170f","Type":"ContainerDied","Data":"b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927"}
May 20 14:30:39.016500 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:39.016465    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96de3b-d9a0-430d-8ce4-20063e730550-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:39.018979 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:39.018959    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96de3b-d9a0-430d-8ce4-20063e730550-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:39.181942 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:39.181918    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:39.309941 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:39.309869    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"]
May 20 14:30:39.313193 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:30:39.313163    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e96de3b_d9a0_430d_8ce4_20063e730550.slice/crio-cc7244b46d3a45cbf27315d510caac3545cef5e1f0932871f8cc2b6229833797 WatchSource:0}: Error finding container cc7244b46d3a45cbf27315d510caac3545cef5e1f0932871f8cc2b6229833797: Status 404 returned error can't find the container with id cc7244b46d3a45cbf27315d510caac3545cef5e1f0932871f8cc2b6229833797
May 20 14:30:39.315069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:39.315053    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:30:39.724529 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:39.724492    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" event={"ID":"6e96de3b-d9a0-430d-8ce4-20063e730550","Type":"ContainerStarted","Data":"a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b"}
May 20 14:30:39.724529 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:39.724533    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" event={"ID":"6e96de3b-d9a0-430d-8ce4-20063e730550","Type":"ContainerStarted","Data":"cc7244b46d3a45cbf27315d510caac3545cef5e1f0932871f8cc2b6229833797"}
May 20 14:30:40.462839 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:40.462798    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
May 20 14:30:40.467092 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:40.467070    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 20 14:30:42.549440 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.549418    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:30:42.642648 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.642616    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/bd356406-964f-49ee-8ae0-115658c0170f-kube-api-access-dw84v\") pod \"bd356406-964f-49ee-8ae0-115658c0170f\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") "
May 20 14:30:42.642808 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.642681    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"bd356406-964f-49ee-8ae0-115658c0170f\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") "
May 20 14:30:42.642808 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.642780    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd356406-964f-49ee-8ae0-115658c0170f-proxy-tls\") pod \"bd356406-964f-49ee-8ae0-115658c0170f\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") "
May 20 14:30:42.643082 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.643051    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "bd356406-964f-49ee-8ae0-115658c0170f" (UID: "bd356406-964f-49ee-8ae0-115658c0170f"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:30:42.644714 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.644685    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd356406-964f-49ee-8ae0-115658c0170f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd356406-964f-49ee-8ae0-115658c0170f" (UID: "bd356406-964f-49ee-8ae0-115658c0170f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:30:42.644802 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.644749    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd356406-964f-49ee-8ae0-115658c0170f-kube-api-access-dw84v" (OuterVolumeSpecName: "kube-api-access-dw84v") pod "bd356406-964f-49ee-8ae0-115658c0170f" (UID: "bd356406-964f-49ee-8ae0-115658c0170f"). InnerVolumeSpecName "kube-api-access-dw84v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:30:42.741525 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.741439    2557 generic.go:358] "Generic (PLEG): container finished" podID="bd356406-964f-49ee-8ae0-115658c0170f" containerID="ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22" exitCode=0
May 20 14:30:42.741657 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.741518    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" event={"ID":"bd356406-964f-49ee-8ae0-115658c0170f","Type":"ContainerDied","Data":"ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22"}
May 20 14:30:42.741657 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.741547    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"
May 20 14:30:42.741657 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.741554    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k" event={"ID":"bd356406-964f-49ee-8ae0-115658c0170f","Type":"ContainerDied","Data":"d031fe80dab4435ffd5c35e3724fc2b067c4b2d15da732f87ca29d5e028a28c3"}
May 20 14:30:42.741657 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.741569    2557 scope.go:117] "RemoveContainer" containerID="b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927"
May 20 14:30:42.743366 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.743345    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-cabundle-cert\") pod \"bd356406-964f-49ee-8ae0-115658c0170f\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") "
May 20 14:30:42.743495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.743386    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd356406-964f-49ee-8ae0-115658c0170f-kserve-provision-location\") pod \"bd356406-964f-49ee-8ae0-115658c0170f\" (UID: \"bd356406-964f-49ee-8ae0-115658c0170f\") "
May 20 14:30:42.743598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.743523    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw84v\" (UniqueName: \"kubernetes.io/projected/bd356406-964f-49ee-8ae0-115658c0170f-kube-api-access-dw84v\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:30:42.743598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.743535    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:30:42.743598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.743545    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd356406-964f-49ee-8ae0-115658c0170f-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:30:42.743763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.743661    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd356406-964f-49ee-8ae0-115658c0170f" (UID: "bd356406-964f-49ee-8ae0-115658c0170f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:30:42.743763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.743707    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd356406-964f-49ee-8ae0-115658c0170f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd356406-964f-49ee-8ae0-115658c0170f" (UID: "bd356406-964f-49ee-8ae0-115658c0170f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:30:42.749762 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.749744    2557 scope.go:117] "RemoveContainer" containerID="ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22"
May 20 14:30:42.756974 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.756959    2557 scope.go:117] "RemoveContainer" containerID="8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764"
May 20 14:30:42.763644 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.763630    2557 scope.go:117] "RemoveContainer" containerID="b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927"
May 20 14:30:42.763917 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:30:42.763891    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927\": container with ID starting with b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927 not found: ID does not exist" containerID="b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927"
May 20 14:30:42.763995 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.763926    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927"} err="failed to get container status \"b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927\": rpc error: code = NotFound desc = could not find container \"b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927\": container with ID starting with b56216bad1241a7172a207e0861a2235da326423285c7b40891a77712a9ff927 not found: ID does not exist"
May 20 14:30:42.763995 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.763951    2557 scope.go:117] "RemoveContainer" containerID="ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22"
May 20 14:30:42.764205 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:30:42.764184    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22\": container with ID starting with ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22 not found: ID does not exist" containerID="ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22"
May 20 14:30:42.764298 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.764208    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22"} err="failed to get container status \"ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22\": rpc error: code = NotFound desc = could not find container \"ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22\": container with ID starting with ede32b48d05fcf0ca2b58e51809acb8752372d6704989d29615e3369a51cbd22 not found: ID does not exist"
May 20 14:30:42.764298 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.764224    2557 scope.go:117] "RemoveContainer" containerID="8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764"
May 20 14:30:42.764455 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:30:42.764439    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764\": container with ID starting with 8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764 not found: ID does not exist" containerID="8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764"
May 20 14:30:42.764500 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.764460    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764"} err="failed to get container status \"8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764\": rpc error: code = NotFound desc = could not find container \"8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764\": container with ID starting with 8707ab811ca7334a076d715ca6e99b727c8b04553fc712a59714699a85ccb764 not found: ID does not exist"
May 20 14:30:42.843885 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.843847    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd356406-964f-49ee-8ae0-115658c0170f-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:30:42.843885 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:42.843883    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd356406-964f-49ee-8ae0-115658c0170f-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:30:43.064972 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:43.064944    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"]
May 20 14:30:43.069044 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:43.069018    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dg94k"]
May 20 14:30:43.747405 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:43.747370    2557 generic.go:358] "Generic (PLEG): container finished" podID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerID="a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b" exitCode=0
May 20 14:30:43.747792 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:43.747425    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" event={"ID":"6e96de3b-d9a0-430d-8ce4-20063e730550","Type":"ContainerDied","Data":"a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b"}
May 20 14:30:44.405403 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:44.405369    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd356406-964f-49ee-8ae0-115658c0170f" path="/var/lib/kubelet/pods/bd356406-964f-49ee-8ae0-115658c0170f/volumes"
May 20 14:30:44.752037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:44.751957    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" event={"ID":"6e96de3b-d9a0-430d-8ce4-20063e730550","Type":"ContainerStarted","Data":"1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31"}
May 20 14:30:44.752037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:44.751992    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" event={"ID":"6e96de3b-d9a0-430d-8ce4-20063e730550","Type":"ContainerStarted","Data":"c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815"}
May 20 14:30:44.752395 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:44.752182    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:44.774485 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:44.774443    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" podStartSLOduration=6.774431804 podStartE2EDuration="6.774431804s" podCreationTimestamp="2026-05-20 14:30:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:30:44.772997893 +0000 UTC m=+2424.910552115" watchObservedRunningTime="2026-05-20 14:30:44.774431804 +0000 UTC m=+2424.911986023"
May 20 14:30:45.754868 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:45.754827    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:30:51.764031 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:30:51.763998    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:31:21.764712 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:31:21.764670    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:31:31.765114 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:31:31.765074    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:31:41.765105 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:31:41.765065    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:31:51.765168 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:31:51.765125    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:32:01.769073 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:01.768967    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:32:08.212593 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.212555    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"]
May 20 14:32:08.213090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.213020    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kserve-container" containerID="cri-o://c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815" gracePeriod=30
May 20 14:32:08.213165 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.213080    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kube-rbac-proxy" containerID="cri-o://1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31" gracePeriod=30
May 20 14:32:08.309128 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.309093    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"]
May 20 14:32:08.309495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.309482    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="storage-initializer"
May 20 14:32:08.309548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.309497    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="storage-initializer"
May 20 14:32:08.309548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.309513    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container"
May 20 14:32:08.309548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.309519    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container"
May 20 14:32:08.309548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.309530    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kube-rbac-proxy"
May 20 14:32:08.309548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.309539    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kube-rbac-proxy"
May 20 14:32:08.309712 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.309600    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kserve-container"
May 20 14:32:08.309712 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.309610    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="bd356406-964f-49ee-8ae0-115658c0170f" containerName="kube-rbac-proxy"
May 20 14:32:08.312978 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.312963    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.315948 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.315927    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
May 20 14:32:08.316056 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.315927    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
May 20 14:32:08.324266 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.324246    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"]
May 20 14:32:08.392718 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.392694    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w98f\" (UniqueName: \"kubernetes.io/projected/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kube-api-access-6w98f\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.392831 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.392742    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2881701a-0d8c-46a5-93d3-f7320b80b5bd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.392831 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.392816    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.392968 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.392904    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.392968 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.392929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.493877 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.493801    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2881701a-0d8c-46a5-93d3-f7320b80b5bd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.493877 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.493848    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.494049 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.493898    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.494049 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.493918    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.494049 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.493949    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6w98f\" (UniqueName: \"kubernetes.io/projected/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kube-api-access-6w98f\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.494394 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.494373    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.494578 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.494558    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.494666 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.494647    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.496431 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.496409    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2881701a-0d8c-46a5-93d3-f7320b80b5bd-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.502651 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.502628    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w98f\" (UniqueName: \"kubernetes.io/projected/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kube-api-access-6w98f\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.623436 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.623414    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:08.749411 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:08.749385    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"]
May 20 14:32:08.751708 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:32:08.751684    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2881701a_0d8c_46a5_93d3_f7320b80b5bd.slice/crio-9ed55d358fd1f56bfd2cd9f871b969e7c21819852ce743e0c6873ed71f86234d WatchSource:0}: Error finding container 9ed55d358fd1f56bfd2cd9f871b969e7c21819852ce743e0c6873ed71f86234d: Status 404 returned error can't find the container with id 9ed55d358fd1f56bfd2cd9f871b969e7c21819852ce743e0c6873ed71f86234d
May 20 14:32:09.022443 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:09.022355    2557 generic.go:358] "Generic (PLEG): container finished" podID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerID="1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31" exitCode=2
May 20 14:32:09.022582 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:09.022439    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" event={"ID":"6e96de3b-d9a0-430d-8ce4-20063e730550","Type":"ContainerDied","Data":"1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31"}
May 20 14:32:09.023817 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:09.023795    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" event={"ID":"2881701a-0d8c-46a5-93d3-f7320b80b5bd","Type":"ContainerStarted","Data":"aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db"}
May 20 14:32:09.023932 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:09.023823    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" event={"ID":"2881701a-0d8c-46a5-93d3-f7320b80b5bd","Type":"ContainerStarted","Data":"9ed55d358fd1f56bfd2cd9f871b969e7c21819852ce743e0c6873ed71f86234d"}
May 20 14:32:11.759078 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:11.759032    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
May 20 14:32:11.765194 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:11.765165    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
May 20 14:32:12.562292 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.562270    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:32:12.628974 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.628949    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jklcf\" (UniqueName: \"kubernetes.io/projected/6e96de3b-d9a0-430d-8ce4-20063e730550-kube-api-access-jklcf\") pod \"6e96de3b-d9a0-430d-8ce4-20063e730550\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") "
May 20 14:32:12.629103 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.629004    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96de3b-d9a0-430d-8ce4-20063e730550-proxy-tls\") pod \"6e96de3b-d9a0-430d-8ce4-20063e730550\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") "
May 20 14:32:12.629103 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.629037    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-cabundle-cert\") pod \"6e96de3b-d9a0-430d-8ce4-20063e730550\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") "
May 20 14:32:12.629103 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.629081    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96de3b-d9a0-430d-8ce4-20063e730550-kserve-provision-location\") pod \"6e96de3b-d9a0-430d-8ce4-20063e730550\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") "
May 20 14:32:12.629225 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.629114    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"6e96de3b-d9a0-430d-8ce4-20063e730550\" (UID: \"6e96de3b-d9a0-430d-8ce4-20063e730550\") "
May 20 14:32:12.629467 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.629442    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e96de3b-d9a0-430d-8ce4-20063e730550-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e96de3b-d9a0-430d-8ce4-20063e730550" (UID: "6e96de3b-d9a0-430d-8ce4-20063e730550"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:32:12.629540 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.629460    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e96de3b-d9a0-430d-8ce4-20063e730550" (UID: "6e96de3b-d9a0-430d-8ce4-20063e730550"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:32:12.629540 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.629472    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "6e96de3b-d9a0-430d-8ce4-20063e730550" (UID: "6e96de3b-d9a0-430d-8ce4-20063e730550"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:32:12.630992 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.630969    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e96de3b-d9a0-430d-8ce4-20063e730550-kube-api-access-jklcf" (OuterVolumeSpecName: "kube-api-access-jklcf") pod "6e96de3b-d9a0-430d-8ce4-20063e730550" (UID: "6e96de3b-d9a0-430d-8ce4-20063e730550"). InnerVolumeSpecName "kube-api-access-jklcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:32:12.631094 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.631031    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e96de3b-d9a0-430d-8ce4-20063e730550-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e96de3b-d9a0-430d-8ce4-20063e730550" (UID: "6e96de3b-d9a0-430d-8ce4-20063e730550"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:32:12.735116 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.730218    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jklcf\" (UniqueName: \"kubernetes.io/projected/6e96de3b-d9a0-430d-8ce4-20063e730550-kube-api-access-jklcf\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:32:12.735116 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.730259    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e96de3b-d9a0-430d-8ce4-20063e730550-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:32:12.735116 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.730277    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:32:12.735116 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.730344    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e96de3b-d9a0-430d-8ce4-20063e730550-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:32:12.735116 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:12.730364    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e96de3b-d9a0-430d-8ce4-20063e730550-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:32:13.038146 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.038040    2557 generic.go:358] "Generic (PLEG): container finished" podID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerID="c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815" exitCode=0
May 20 14:32:13.038591 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.038130    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" event={"ID":"6e96de3b-d9a0-430d-8ce4-20063e730550","Type":"ContainerDied","Data":"c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815"}
May 20 14:32:13.038591 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.038160    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"
May 20 14:32:13.038591 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.038184    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6" event={"ID":"6e96de3b-d9a0-430d-8ce4-20063e730550","Type":"ContainerDied","Data":"cc7244b46d3a45cbf27315d510caac3545cef5e1f0932871f8cc2b6229833797"}
May 20 14:32:13.038591 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.038201    2557 scope.go:117] "RemoveContainer" containerID="1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31"
May 20 14:32:13.039548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.039513    2557 generic.go:358] "Generic (PLEG): container finished" podID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerID="aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db" exitCode=0
May 20 14:32:13.039657 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.039581    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" event={"ID":"2881701a-0d8c-46a5-93d3-f7320b80b5bd","Type":"ContainerDied","Data":"aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db"}
May 20 14:32:13.047008 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.046897    2557 scope.go:117] "RemoveContainer" containerID="c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815"
May 20 14:32:13.054760 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.054743    2557 scope.go:117] "RemoveContainer" containerID="a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b"
May 20 14:32:13.062225 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.062210    2557 scope.go:117] "RemoveContainer" containerID="1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31"
May 20 14:32:13.062474 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:32:13.062455    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31\": container with ID starting with 1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31 not found: ID does not exist" containerID="1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31"
May 20 14:32:13.062529 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.062482    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31"} err="failed to get container status \"1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31\": rpc error: code = NotFound desc = could not find container \"1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31\": container with ID starting with 1052be57b228ab9b4970c68ff694a312bc95cd53f72229f7250b97a88ca10f31 not found: ID does not exist"
May 20 14:32:13.062529 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.062499    2557 scope.go:117] "RemoveContainer" containerID="c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815"
May 20 14:32:13.062737 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:32:13.062718    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815\": container with ID starting with c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815 not found: ID does not exist" containerID="c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815"
May 20 14:32:13.062778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.062743    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815"} err="failed to get container status \"c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815\": rpc error: code = NotFound desc = could not find container \"c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815\": container with ID starting with c7b72d723c0d31226849dbe1708b70d1e52b030cc0846580d4777caaba8d3815 not found: ID does not exist"
May 20 14:32:13.062778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.062760    2557 scope.go:117] "RemoveContainer" containerID="a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b"
May 20 14:32:13.063007 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:32:13.062988    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b\": container with ID starting with a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b not found: ID does not exist" containerID="a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b"
May 20 14:32:13.063058 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.063011    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b"} err="failed to get container status \"a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b\": rpc error: code = NotFound desc = could not find container \"a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b\": container with ID starting with a52395ee587039f11bf4592e701cdf6dd6b73a3aeda8cf48f738db1b3fb0994b not found: ID does not exist"
May 20 14:32:13.081236 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.081210    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"]
May 20 14:32:13.085439 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:13.085417    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-ztgr6"]
May 20 14:32:14.045984 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:14.045949    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" event={"ID":"2881701a-0d8c-46a5-93d3-f7320b80b5bd","Type":"ContainerStarted","Data":"ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61"}
May 20 14:32:14.046461 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:14.045988    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" event={"ID":"2881701a-0d8c-46a5-93d3-f7320b80b5bd","Type":"ContainerStarted","Data":"4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f"}
May 20 14:32:14.046461 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:14.046209    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:14.067807 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:14.067765    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" podStartSLOduration=6.067753896 podStartE2EDuration="6.067753896s" podCreationTimestamp="2026-05-20 14:32:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:32:14.066604398 +0000 UTC m=+2514.204158618" watchObservedRunningTime="2026-05-20 14:32:14.067753896 +0000 UTC m=+2514.205308115"
May 20 14:32:14.405678 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:14.405649    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" path="/var/lib/kubelet/pods/6e96de3b-d9a0-430d-8ce4-20063e730550/volumes"
May 20 14:32:15.049000 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:15.048973    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:21.058128 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:21.057457    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:32:51.059424 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:32:51.059384    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
May 20 14:33:01.058918 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:01.058876    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
May 20 14:33:11.058617 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:11.058581    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
May 20 14:33:21.058710 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:21.058673    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
May 20 14:33:31.061982 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:31.061895    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:33:38.467108 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.467066    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"]
May 20 14:33:38.467595 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.467384    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kserve-container" containerID="cri-o://4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f" gracePeriod=30
May 20 14:33:38.467681 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.467575    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kube-rbac-proxy" containerID="cri-o://ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61" gracePeriod=30
May 20 14:33:38.593352 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.593326    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"]
May 20 14:33:38.593680 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.593667    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kserve-container"
May 20 14:33:38.593723 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.593681    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kserve-container"
May 20 14:33:38.593723 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.593689    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kube-rbac-proxy"
May 20 14:33:38.593723 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.593695    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kube-rbac-proxy"
May 20 14:33:38.593723 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.593703    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="storage-initializer"
May 20 14:33:38.593723 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.593708    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="storage-initializer"
May 20 14:33:38.593933 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.593761    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kserve-container"
May 20 14:33:38.593933 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.593773    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="6e96de3b-d9a0-430d-8ce4-20063e730550" containerName="kube-rbac-proxy"
May 20 14:33:38.596817 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.596801    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.599540 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.599523    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
May 20 14:33:38.599761 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.599746    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
May 20 14:33:38.607603 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.607582    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"]
May 20 14:33:38.671153 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.671127    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7069de9-7035-4c8b-ae79-b0683765f201-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.671263 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.671162    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.671263 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.671202    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxtdl\" (UniqueName: \"kubernetes.io/projected/a7069de9-7035-4c8b-ae79-b0683765f201-kube-api-access-jxtdl\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.671263 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.671222    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7069de9-7035-4c8b-ae79-b0683765f201-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.671263 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.671239    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.772192 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.772139    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtdl\" (UniqueName: \"kubernetes.io/projected/a7069de9-7035-4c8b-ae79-b0683765f201-kube-api-access-jxtdl\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.772192 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.772168    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7069de9-7035-4c8b-ae79-b0683765f201-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.772192 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.772190    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.772412 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.772226    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7069de9-7035-4c8b-ae79-b0683765f201-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.772412 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.772251    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.772720 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.772695    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7069de9-7035-4c8b-ae79-b0683765f201-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.772823 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.772806    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.772965 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.772947    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.774483 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.774462    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7069de9-7035-4c8b-ae79-b0683765f201-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.782605 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.782584    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtdl\" (UniqueName: \"kubernetes.io/projected/a7069de9-7035-4c8b-ae79-b0683765f201-kube-api-access-jxtdl\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:38.908230 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:38.908206    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:39.044755 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:39.044732    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"]
May 20 14:33:39.047021 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:33:39.046990    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7069de9_7035_4c8b_ae79_b0683765f201.slice/crio-6c92f1174405a5b29b1b4bc841789a7c032a1217a07f493dbfc908a63200a029 WatchSource:0}: Error finding container 6c92f1174405a5b29b1b4bc841789a7c032a1217a07f493dbfc908a63200a029: Status 404 returned error can't find the container with id 6c92f1174405a5b29b1b4bc841789a7c032a1217a07f493dbfc908a63200a029
May 20 14:33:39.303183 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:39.303087    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" event={"ID":"a7069de9-7035-4c8b-ae79-b0683765f201","Type":"ContainerStarted","Data":"2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d"}
May 20 14:33:39.303183 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:39.303138    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" event={"ID":"a7069de9-7035-4c8b-ae79-b0683765f201","Type":"ContainerStarted","Data":"6c92f1174405a5b29b1b4bc841789a7c032a1217a07f493dbfc908a63200a029"}
May 20 14:33:39.305022 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:39.304996    2557 generic.go:358] "Generic (PLEG): container finished" podID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerID="ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61" exitCode=2
May 20 14:33:39.305151 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:39.305048    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" event={"ID":"2881701a-0d8c-46a5-93d3-f7320b80b5bd","Type":"ContainerDied","Data":"ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61"}
May 20 14:33:41.052430 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:41.052384    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
May 20 14:33:41.058892 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:41.058833    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
May 20 14:33:42.512301 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.512280    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:33:42.603146 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.603123    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-cabundle-cert\") pod \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") "
May 20 14:33:42.603289 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.603182    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w98f\" (UniqueName: \"kubernetes.io/projected/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kube-api-access-6w98f\") pod \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") "
May 20 14:33:42.603289 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.603212    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") "
May 20 14:33:42.603289 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.603275    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2881701a-0d8c-46a5-93d3-f7320b80b5bd-proxy-tls\") pod \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") "
May 20 14:33:42.603444 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.603409    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kserve-provision-location\") pod \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\" (UID: \"2881701a-0d8c-46a5-93d3-f7320b80b5bd\") "
May 20 14:33:42.603594 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.603569    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2881701a-0d8c-46a5-93d3-f7320b80b5bd" (UID: "2881701a-0d8c-46a5-93d3-f7320b80b5bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:33:42.603753 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.603731    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2881701a-0d8c-46a5-93d3-f7320b80b5bd" (UID: "2881701a-0d8c-46a5-93d3-f7320b80b5bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:33:42.603814 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.603752    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:33:42.603814 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.603766    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "2881701a-0d8c-46a5-93d3-f7320b80b5bd" (UID: "2881701a-0d8c-46a5-93d3-f7320b80b5bd"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:33:42.605352 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.605321    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kube-api-access-6w98f" (OuterVolumeSpecName: "kube-api-access-6w98f") pod "2881701a-0d8c-46a5-93d3-f7320b80b5bd" (UID: "2881701a-0d8c-46a5-93d3-f7320b80b5bd"). InnerVolumeSpecName "kube-api-access-6w98f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:33:42.605446 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.605408    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2881701a-0d8c-46a5-93d3-f7320b80b5bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2881701a-0d8c-46a5-93d3-f7320b80b5bd" (UID: "2881701a-0d8c-46a5-93d3-f7320b80b5bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:33:42.704741 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.704719    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:33:42.704741 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.704741    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w98f\" (UniqueName: \"kubernetes.io/projected/2881701a-0d8c-46a5-93d3-f7320b80b5bd-kube-api-access-6w98f\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:33:42.704910 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.704754    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2881701a-0d8c-46a5-93d3-f7320b80b5bd-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:33:42.704910 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:42.704765    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2881701a-0d8c-46a5-93d3-f7320b80b5bd-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:33:43.318990 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.318962    2557 generic.go:358] "Generic (PLEG): container finished" podID="a7069de9-7035-4c8b-ae79-b0683765f201" containerID="2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d" exitCode=0
May 20 14:33:43.319117 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.319042    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" event={"ID":"a7069de9-7035-4c8b-ae79-b0683765f201","Type":"ContainerDied","Data":"2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d"}
May 20 14:33:43.320787 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.320765    2557 generic.go:358] "Generic (PLEG): container finished" podID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerID="4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f" exitCode=0
May 20 14:33:43.320912 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.320808    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" event={"ID":"2881701a-0d8c-46a5-93d3-f7320b80b5bd","Type":"ContainerDied","Data":"4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f"}
May 20 14:33:43.320912 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.320827    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8" event={"ID":"2881701a-0d8c-46a5-93d3-f7320b80b5bd","Type":"ContainerDied","Data":"9ed55d358fd1f56bfd2cd9f871b969e7c21819852ce743e0c6873ed71f86234d"}
May 20 14:33:43.320912 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.320828    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"
May 20 14:33:43.320912 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.320868    2557 scope.go:117] "RemoveContainer" containerID="ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61"
May 20 14:33:43.329635 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.329620    2557 scope.go:117] "RemoveContainer" containerID="4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f"
May 20 14:33:43.336756 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.336740    2557 scope.go:117] "RemoveContainer" containerID="aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db"
May 20 14:33:43.348442 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.348425    2557 scope.go:117] "RemoveContainer" containerID="ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61"
May 20 14:33:43.348699 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:33:43.348682    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61\": container with ID starting with ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61 not found: ID does not exist" containerID="ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61"
May 20 14:33:43.348787 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.348713    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61"} err="failed to get container status \"ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61\": rpc error: code = NotFound desc = could not find container \"ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61\": container with ID starting with ea74458b277485f672f84b96849c2a25e1a1887fb063c230455c4908ad8b0d61 not found: ID does not exist"
May 20 14:33:43.348787 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.348736    2557 scope.go:117] "RemoveContainer" containerID="4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f"
May 20 14:33:43.349017 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:33:43.348999    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f\": container with ID starting with 4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f not found: ID does not exist" containerID="4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f"
May 20 14:33:43.349070 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.349022    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f"} err="failed to get container status \"4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f\": rpc error: code = NotFound desc = could not find container \"4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f\": container with ID starting with 4922f7c113373c8138be89c9ea177f4a1ad03445ca02a2c4cd8edc257017f65f not found: ID does not exist"
May 20 14:33:43.349070 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.349039    2557 scope.go:117] "RemoveContainer" containerID="aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db"
May 20 14:33:43.349255 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:33:43.349235    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db\": container with ID starting with aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db not found: ID does not exist" containerID="aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db"
May 20 14:33:43.349354 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.349255    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db"} err="failed to get container status \"aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db\": rpc error: code = NotFound desc = could not find container \"aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db\": container with ID starting with aaa027989e429aa22808f600760e50689b63653fabe88cf74366634103a020db not found: ID does not exist"
May 20 14:33:43.358442 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.358415    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"]
May 20 14:33:43.364884 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:43.364848    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-wf5r8"]
May 20 14:33:44.329339 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:44.329306    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" event={"ID":"a7069de9-7035-4c8b-ae79-b0683765f201","Type":"ContainerStarted","Data":"f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3"}
May 20 14:33:44.329767 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:44.329349    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" event={"ID":"a7069de9-7035-4c8b-ae79-b0683765f201","Type":"ContainerStarted","Data":"c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068"}
May 20 14:33:44.329767 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:44.329554    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:44.357687 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:44.357644    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" podStartSLOduration=6.357631792 podStartE2EDuration="6.357631792s" podCreationTimestamp="2026-05-20 14:33:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:33:44.354978941 +0000 UTC m=+2604.492533160" watchObservedRunningTime="2026-05-20 14:33:44.357631792 +0000 UTC m=+2604.495186011"
May 20 14:33:44.405112 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:44.405093    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" path="/var/lib/kubelet/pods/2881701a-0d8c-46a5-93d3-f7320b80b5bd/volumes"
May 20 14:33:45.332551 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:45.332526    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:33:51.340921 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:33:51.340892    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:34:21.342097 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:34:21.342060    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:34:31.341590 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:34:31.341547    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:34:41.341525 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:34:41.341486    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:34:51.345029 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:34:51.344993    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:34:58.620314 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:34:58.620264    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"]
May 20 14:34:58.620773 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:34:58.620712    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kserve-container" containerID="cri-o://c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068" gracePeriod=30
May 20 14:34:58.621067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:34:58.620795    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kube-rbac-proxy" containerID="cri-o://f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3" gracePeriod=30
May 20 14:34:59.562135 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:34:59.562096    2557 generic.go:358] "Generic (PLEG): container finished" podID="a7069de9-7035-4c8b-ae79-b0683765f201" containerID="f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3" exitCode=2
May 20 14:34:59.562296 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:34:59.562173    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" event={"ID":"a7069de9-7035-4c8b-ae79-b0683765f201","Type":"ContainerDied","Data":"f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3"}
May 20 14:35:00.920091 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.920017    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"]
May 20 14:35:00.920478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.920370    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kserve-container"
May 20 14:35:00.920478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.920383    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kserve-container"
May 20 14:35:00.920478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.920398    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="storage-initializer"
May 20 14:35:00.920478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.920406    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="storage-initializer"
May 20 14:35:00.920478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.920421    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kube-rbac-proxy"
May 20 14:35:00.920478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.920429    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kube-rbac-proxy"
May 20 14:35:00.920801 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.920506    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kube-rbac-proxy"
May 20 14:35:00.920801 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.920520    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="2881701a-0d8c-46a5-93d3-f7320b80b5bd" containerName="kserve-container"
May 20 14:35:00.923715 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.923697    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:00.927447 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.927420    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
May 20 14:35:00.927743 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.927728    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
May 20 14:35:00.944833 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.944807    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"]
May 20 14:35:00.974364 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.974341    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:00.974469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.974383    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-cabundle-cert\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:00.974469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.974413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5229a93-9882-4b81-8d8b-21bd33a3951f-kserve-provision-location\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:00.974469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.974457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5229a93-9882-4b81-8d8b-21bd33a3951f-proxy-tls\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:00.974606 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:00.974492    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nx7f\" (UniqueName: \"kubernetes.io/projected/b5229a93-9882-4b81-8d8b-21bd33a3951f-kube-api-access-8nx7f\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.075364 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.075340    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.075460 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.075385    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-cabundle-cert\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.075460 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.075409    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5229a93-9882-4b81-8d8b-21bd33a3951f-kserve-provision-location\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.075588 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.075520    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5229a93-9882-4b81-8d8b-21bd33a3951f-proxy-tls\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.075644 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.075590    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8nx7f\" (UniqueName: \"kubernetes.io/projected/b5229a93-9882-4b81-8d8b-21bd33a3951f-kube-api-access-8nx7f\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.075750 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.075732    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5229a93-9882-4b81-8d8b-21bd33a3951f-kserve-provision-location\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.076074 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.076053    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-cabundle-cert\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.076149 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.076074    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.077847 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.077827    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5229a93-9882-4b81-8d8b-21bd33a3951f-proxy-tls\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.086938 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.086914    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nx7f\" (UniqueName: \"kubernetes.io/projected/b5229a93-9882-4b81-8d8b-21bd33a3951f-kube-api-access-8nx7f\") pod \"isvc-sklearn-predictor-76db96c987-bb7gr\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.236683 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.236619    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:01.336427 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.336392    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
May 20 14:35:01.341956 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.341931    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
May 20 14:35:01.389348 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.389327    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"]
May 20 14:35:01.392192 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:35:01.392168    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5229a93_9882_4b81_8d8b_21bd33a3951f.slice/crio-d3b03bec2d62b7122bac269eff9333dda94920ed82399c15451ff9b667a93802 WatchSource:0}: Error finding container d3b03bec2d62b7122bac269eff9333dda94920ed82399c15451ff9b667a93802: Status 404 returned error can't find the container with id d3b03bec2d62b7122bac269eff9333dda94920ed82399c15451ff9b667a93802
May 20 14:35:01.569502 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.569465    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" event={"ID":"b5229a93-9882-4b81-8d8b-21bd33a3951f","Type":"ContainerStarted","Data":"16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1"}
May 20 14:35:01.569678 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:01.569514    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" event={"ID":"b5229a93-9882-4b81-8d8b-21bd33a3951f","Type":"ContainerStarted","Data":"d3b03bec2d62b7122bac269eff9333dda94920ed82399c15451ff9b667a93802"}
May 20 14:35:03.163261 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.163236    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:35:03.293122 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.293050    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxtdl\" (UniqueName: \"kubernetes.io/projected/a7069de9-7035-4c8b-ae79-b0683765f201-kube-api-access-jxtdl\") pod \"a7069de9-7035-4c8b-ae79-b0683765f201\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") "
May 20 14:35:03.293122 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.293112    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7069de9-7035-4c8b-ae79-b0683765f201-proxy-tls\") pod \"a7069de9-7035-4c8b-ae79-b0683765f201\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") "
May 20 14:35:03.293283 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.293141    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7069de9-7035-4c8b-ae79-b0683765f201-kserve-provision-location\") pod \"a7069de9-7035-4c8b-ae79-b0683765f201\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") "
May 20 14:35:03.293283 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.293195    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"a7069de9-7035-4c8b-ae79-b0683765f201\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") "
May 20 14:35:03.293283 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.293230    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-cabundle-cert\") pod \"a7069de9-7035-4c8b-ae79-b0683765f201\" (UID: \"a7069de9-7035-4c8b-ae79-b0683765f201\") "
May 20 14:35:03.293609 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.293583    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "a7069de9-7035-4c8b-ae79-b0683765f201" (UID: "a7069de9-7035-4c8b-ae79-b0683765f201"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:35:03.293708 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.293585    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7069de9-7035-4c8b-ae79-b0683765f201-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7069de9-7035-4c8b-ae79-b0683765f201" (UID: "a7069de9-7035-4c8b-ae79-b0683765f201"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:35:03.293708 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.293636    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7069de9-7035-4c8b-ae79-b0683765f201" (UID: "a7069de9-7035-4c8b-ae79-b0683765f201"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:35:03.295165 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.295146    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7069de9-7035-4c8b-ae79-b0683765f201-kube-api-access-jxtdl" (OuterVolumeSpecName: "kube-api-access-jxtdl") pod "a7069de9-7035-4c8b-ae79-b0683765f201" (UID: "a7069de9-7035-4c8b-ae79-b0683765f201"). InnerVolumeSpecName "kube-api-access-jxtdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:35:03.295277 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.295237    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7069de9-7035-4c8b-ae79-b0683765f201-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7069de9-7035-4c8b-ae79-b0683765f201" (UID: "a7069de9-7035-4c8b-ae79-b0683765f201"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:35:03.393837 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.393813    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7069de9-7035-4c8b-ae79-b0683765f201-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:35:03.393837 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.393835    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7069de9-7035-4c8b-ae79-b0683765f201-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:35:03.393988 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.393847    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:35:03.393988 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.393887    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7069de9-7035-4c8b-ae79-b0683765f201-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:35:03.393988 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.393897    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxtdl\" (UniqueName: \"kubernetes.io/projected/a7069de9-7035-4c8b-ae79-b0683765f201-kube-api-access-jxtdl\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:35:03.576642 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.576614    2557 generic.go:358] "Generic (PLEG): container finished" podID="a7069de9-7035-4c8b-ae79-b0683765f201" containerID="c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068" exitCode=0
May 20 14:35:03.576750 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.576666    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" event={"ID":"a7069de9-7035-4c8b-ae79-b0683765f201","Type":"ContainerDied","Data":"c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068"}
May 20 14:35:03.576750 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.576687    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"
May 20 14:35:03.576750 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.576708    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9" event={"ID":"a7069de9-7035-4c8b-ae79-b0683765f201","Type":"ContainerDied","Data":"6c92f1174405a5b29b1b4bc841789a7c032a1217a07f493dbfc908a63200a029"}
May 20 14:35:03.576750 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.576725    2557 scope.go:117] "RemoveContainer" containerID="f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3"
May 20 14:35:03.586646 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.586627    2557 scope.go:117] "RemoveContainer" containerID="c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068"
May 20 14:35:03.594528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.594510    2557 scope.go:117] "RemoveContainer" containerID="2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d"
May 20 14:35:03.602929 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.602909    2557 scope.go:117] "RemoveContainer" containerID="f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3"
May 20 14:35:03.603146 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.603125    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"]
May 20 14:35:03.603218 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:35:03.603180    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3\": container with ID starting with f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3 not found: ID does not exist" containerID="f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3"
May 20 14:35:03.603218 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.603204    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3"} err="failed to get container status \"f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3\": rpc error: code = NotFound desc = could not find container \"f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3\": container with ID starting with f26e16cd9cb619b9e0b7208fe41de269cf47f07e0fd610ba461c716a068d7ba3 not found: ID does not exist"
May 20 14:35:03.603303 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.603221    2557 scope.go:117] "RemoveContainer" containerID="c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068"
May 20 14:35:03.603434 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:35:03.603418    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068\": container with ID starting with c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068 not found: ID does not exist" containerID="c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068"
May 20 14:35:03.603494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.603438    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068"} err="failed to get container status \"c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068\": rpc error: code = NotFound desc = could not find container \"c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068\": container with ID starting with c59ff2a4fcfbb8ab6dbd64b6488cc60b214d1fbfb8406fb9ba737b3e71433068 not found: ID does not exist"
May 20 14:35:03.603494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.603453    2557 scope.go:117] "RemoveContainer" containerID="2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d"
May 20 14:35:03.603705 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:35:03.603686    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d\": container with ID starting with 2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d not found: ID does not exist" containerID="2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d"
May 20 14:35:03.603744 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.603710    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d"} err="failed to get container status \"2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d\": rpc error: code = NotFound desc = could not find container \"2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d\": container with ID starting with 2457d57c7105d422ec8ba3ee0adc3b4938de61f812f15dd323be25f3c031595d not found: ID does not exist"
May 20 14:35:03.608545 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:03.608524    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-rlmz9"]
May 20 14:35:04.405196 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:04.405166    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" path="/var/lib/kubelet/pods/a7069de9-7035-4c8b-ae79-b0683765f201/volumes"
May 20 14:35:05.585575 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:05.585544    2557 generic.go:358] "Generic (PLEG): container finished" podID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerID="16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1" exitCode=0
May 20 14:35:05.585937 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:05.585623    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" event={"ID":"b5229a93-9882-4b81-8d8b-21bd33a3951f","Type":"ContainerDied","Data":"16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1"}
May 20 14:35:06.590045 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:06.590011    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" event={"ID":"b5229a93-9882-4b81-8d8b-21bd33a3951f","Type":"ContainerStarted","Data":"5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682"}
May 20 14:35:06.590418 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:06.590054    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" event={"ID":"b5229a93-9882-4b81-8d8b-21bd33a3951f","Type":"ContainerStarted","Data":"385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c"}
May 20 14:35:06.590418 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:06.590265    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:06.615365 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:06.615252    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podStartSLOduration=6.615240823 podStartE2EDuration="6.615240823s" podCreationTimestamp="2026-05-20 14:35:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:35:06.612201824 +0000 UTC m=+2686.749756043" watchObservedRunningTime="2026-05-20 14:35:06.615240823 +0000 UTC m=+2686.752795041"
May 20 14:35:07.594056 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:07.594026    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:07.595381 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:07.595353    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 20 14:35:08.597256 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:08.597195    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 20 14:35:13.601787 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:13.601760    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:35:13.602252 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:13.602222    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 20 14:35:20.527261 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:20.527233    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:35:20.530799 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:20.530773    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:35:20.531971 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:20.531953    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:35:20.535091 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:20.535076    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:35:23.602465 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:23.602429    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 20 14:35:33.602840 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:33.602804    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 20 14:35:43.603090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:43.603051    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 20 14:35:53.602473 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:35:53.602434    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 20 14:36:03.602385 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:03.602342    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 20 14:36:13.603037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:13.603004    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:36:20.949459 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:20.949417    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"]
May 20 14:36:20.949900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:20.949830    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" containerID="cri-o://385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c" gracePeriod=30
May 20 14:36:20.949990 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:20.949897    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kube-rbac-proxy" containerID="cri-o://5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682" gracePeriod=30
May 20 14:36:21.061275 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.061242    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"]
May 20 14:36:21.061738 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.061720    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kube-rbac-proxy"
May 20 14:36:21.061790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.061740    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kube-rbac-proxy"
May 20 14:36:21.061790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.061758    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="storage-initializer"
May 20 14:36:21.061790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.061764    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="storage-initializer"
May 20 14:36:21.061790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.061781    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kserve-container"
May 20 14:36:21.061790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.061786    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kserve-container"
May 20 14:36:21.061997 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.061841    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kserve-container"
May 20 14:36:21.061997 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.061850    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="a7069de9-7035-4c8b-ae79-b0683765f201" containerName="kube-rbac-proxy"
May 20 14:36:21.064934 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.064915    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.067933 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.067915    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
May 20 14:36:21.068177 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.068162    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
May 20 14:36:21.091262 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.091220    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"]
May 20 14:36:21.169602 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.169566    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.169602 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.169601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rm4\" (UniqueName: \"kubernetes.io/projected/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kube-api-access-f6rm4\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.169816 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.169626    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.169816 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.169684    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.169816 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.169796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.270756 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.270663    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.270756 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.270702    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rm4\" (UniqueName: \"kubernetes.io/projected/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kube-api-access-f6rm4\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.270756 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.270727    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.270756 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.270749    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.271131 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.270800    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.271300 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.271276    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.271455 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.271433    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.271503 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.271444    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.273114 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.273097    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.279971 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.279952    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rm4\" (UniqueName: \"kubernetes.io/projected/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kube-api-access-f6rm4\") pod \"sklearn-v2-mlserver-predictor-65d8664766-zbrpk\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.375608 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.375586    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:21.502565 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.502521    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"]
May 20 14:36:21.504720 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:36:21.504678    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1d10dc_4d4c_4f1d_af1a_670e975b4ac8.slice/crio-318393c251746e99ae9733344d763e392714bd59f14c14eb972c29e7f5155439 WatchSource:0}: Error finding container 318393c251746e99ae9733344d763e392714bd59f14c14eb972c29e7f5155439: Status 404 returned error can't find the container with id 318393c251746e99ae9733344d763e392714bd59f14c14eb972c29e7f5155439
May 20 14:36:21.506576 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.506558    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:36:21.834387 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.834354    2557 generic.go:358] "Generic (PLEG): container finished" podID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerID="5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682" exitCode=2
May 20 14:36:21.834544 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.834430    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" event={"ID":"b5229a93-9882-4b81-8d8b-21bd33a3951f","Type":"ContainerDied","Data":"5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682"}
May 20 14:36:21.835731 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.835706    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" event={"ID":"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8","Type":"ContainerStarted","Data":"2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38"}
May 20 14:36:21.835839 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:21.835735    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" event={"ID":"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8","Type":"ContainerStarted","Data":"318393c251746e99ae9733344d763e392714bd59f14c14eb972c29e7f5155439"}
May 20 14:36:23.598055 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:23.597995    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
May 20 14:36:23.602345 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:23.602314    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 20 14:36:25.287288 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.287264    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:36:25.405023 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.404963    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5229a93-9882-4b81-8d8b-21bd33a3951f-kserve-provision-location\") pod \"b5229a93-9882-4b81-8d8b-21bd33a3951f\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") "
May 20 14:36:25.405023 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.404998    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5229a93-9882-4b81-8d8b-21bd33a3951f-proxy-tls\") pod \"b5229a93-9882-4b81-8d8b-21bd33a3951f\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") "
May 20 14:36:25.405219 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.405027    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"b5229a93-9882-4b81-8d8b-21bd33a3951f\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") "
May 20 14:36:25.405219 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.405051    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-cabundle-cert\") pod \"b5229a93-9882-4b81-8d8b-21bd33a3951f\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") "
May 20 14:36:25.405219 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.405080    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nx7f\" (UniqueName: \"kubernetes.io/projected/b5229a93-9882-4b81-8d8b-21bd33a3951f-kube-api-access-8nx7f\") pod \"b5229a93-9882-4b81-8d8b-21bd33a3951f\" (UID: \"b5229a93-9882-4b81-8d8b-21bd33a3951f\") "
May 20 14:36:25.405392 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.405262    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5229a93-9882-4b81-8d8b-21bd33a3951f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5229a93-9882-4b81-8d8b-21bd33a3951f" (UID: "b5229a93-9882-4b81-8d8b-21bd33a3951f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:36:25.405444 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.405397    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "b5229a93-9882-4b81-8d8b-21bd33a3951f" (UID: "b5229a93-9882-4b81-8d8b-21bd33a3951f"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:36:25.405444 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.405412    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5229a93-9882-4b81-8d8b-21bd33a3951f" (UID: "b5229a93-9882-4b81-8d8b-21bd33a3951f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:36:25.407038 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.407017    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5229a93-9882-4b81-8d8b-21bd33a3951f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5229a93-9882-4b81-8d8b-21bd33a3951f" (UID: "b5229a93-9882-4b81-8d8b-21bd33a3951f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:36:25.407118 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.407101    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5229a93-9882-4b81-8d8b-21bd33a3951f-kube-api-access-8nx7f" (OuterVolumeSpecName: "kube-api-access-8nx7f") pod "b5229a93-9882-4b81-8d8b-21bd33a3951f" (UID: "b5229a93-9882-4b81-8d8b-21bd33a3951f"). InnerVolumeSpecName "kube-api-access-8nx7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:36:25.506196 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.506167    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:36:25.506196 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.506192    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nx7f\" (UniqueName: \"kubernetes.io/projected/b5229a93-9882-4b81-8d8b-21bd33a3951f-kube-api-access-8nx7f\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:36:25.506370 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.506202    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5229a93-9882-4b81-8d8b-21bd33a3951f-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:36:25.506370 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.506212    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5229a93-9882-4b81-8d8b-21bd33a3951f-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:36:25.506370 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.506222    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5229a93-9882-4b81-8d8b-21bd33a3951f-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:36:25.856540 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.856505    2557 generic.go:358] "Generic (PLEG): container finished" podID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerID="385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c" exitCode=0
May 20 14:36:25.856719 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.856590    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"
May 20 14:36:25.856719 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.856584    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" event={"ID":"b5229a93-9882-4b81-8d8b-21bd33a3951f","Type":"ContainerDied","Data":"385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c"}
May 20 14:36:25.856880 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.856739    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr" event={"ID":"b5229a93-9882-4b81-8d8b-21bd33a3951f","Type":"ContainerDied","Data":"d3b03bec2d62b7122bac269eff9333dda94920ed82399c15451ff9b667a93802"}
May 20 14:36:25.856880 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.856766    2557 scope.go:117] "RemoveContainer" containerID="5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682"
May 20 14:36:25.858260 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.858237    2557 generic.go:358] "Generic (PLEG): container finished" podID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerID="2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38" exitCode=0
May 20 14:36:25.858402 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.858271    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" event={"ID":"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8","Type":"ContainerDied","Data":"2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38"}
May 20 14:36:25.865759 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.865745    2557 scope.go:117] "RemoveContainer" containerID="385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c"
May 20 14:36:25.876898 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.876879    2557 scope.go:117] "RemoveContainer" containerID="16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1"
May 20 14:36:25.884391 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.884367    2557 scope.go:117] "RemoveContainer" containerID="5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682"
May 20 14:36:25.884654 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:36:25.884629    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682\": container with ID starting with 5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682 not found: ID does not exist" containerID="5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682"
May 20 14:36:25.884710 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.884655    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682"} err="failed to get container status \"5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682\": rpc error: code = NotFound desc = could not find container \"5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682\": container with ID starting with 5a0cc10351e81314176dc9649a69587b8a4c70413884187e13edbacad815e682 not found: ID does not exist"
May 20 14:36:25.884710 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.884672    2557 scope.go:117] "RemoveContainer" containerID="385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c"
May 20 14:36:25.884975 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:36:25.884957    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c\": container with ID starting with 385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c not found: ID does not exist" containerID="385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c"
May 20 14:36:25.885064 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.884980    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c"} err="failed to get container status \"385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c\": rpc error: code = NotFound desc = could not find container \"385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c\": container with ID starting with 385bb4a5d324c1b8c705b8493e907bcf8fe5dfad59a1271b878e068e6d85187c not found: ID does not exist"
May 20 14:36:25.885064 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.884996    2557 scope.go:117] "RemoveContainer" containerID="16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1"
May 20 14:36:25.885276 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:36:25.885258    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1\": container with ID starting with 16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1 not found: ID does not exist" containerID="16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1"
May 20 14:36:25.885319 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.885281    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1"} err="failed to get container status \"16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1\": rpc error: code = NotFound desc = could not find container \"16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1\": container with ID starting with 16dba4f295d23fa451ca1b4726d08163a2343a7fe539a3a9f6040e957cc40bf1 not found: ID does not exist"
May 20 14:36:25.901875 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.901842    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"]
May 20 14:36:25.906343 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:25.906318    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-76db96c987-bb7gr"]
May 20 14:36:26.406237 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:26.406211    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" path="/var/lib/kubelet/pods/b5229a93-9882-4b81-8d8b-21bd33a3951f/volumes"
May 20 14:36:26.863929 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:26.863889    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" event={"ID":"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8","Type":"ContainerStarted","Data":"85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9"}
May 20 14:36:26.864106 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:26.863934    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" event={"ID":"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8","Type":"ContainerStarted","Data":"a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc"}
May 20 14:36:26.864308 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:26.864286    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:26.864451 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:26.864314    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:36:26.889732 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:26.889689    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" podStartSLOduration=5.889676822 podStartE2EDuration="5.889676822s" podCreationTimestamp="2026-05-20 14:36:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:36:26.888291703 +0000 UTC m=+2767.025845923" watchObservedRunningTime="2026-05-20 14:36:26.889676822 +0000 UTC m=+2767.027231040"
May 20 14:36:32.872885 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:36:32.872842    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:37:02.889819 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:02.889778    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 20 14:37:12.875578 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:12.875542    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:37:21.095215 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.095183    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"]
May 20 14:37:21.095682 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.095511    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kserve-container" containerID="cri-o://a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc" gracePeriod=30
May 20 14:37:21.095682 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.095530    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kube-rbac-proxy" containerID="cri-o://85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9" gracePeriod=30
May 20 14:37:21.196357 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.196323    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"]
May 20 14:37:21.196711 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.196696    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container"
May 20 14:37:21.196711 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.196709    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container"
May 20 14:37:21.196909 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.196725    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kube-rbac-proxy"
May 20 14:37:21.196909 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.196731    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kube-rbac-proxy"
May 20 14:37:21.196909 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.196744    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="storage-initializer"
May 20 14:37:21.196909 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.196749    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="storage-initializer"
May 20 14:37:21.196909 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.196828    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kserve-container"
May 20 14:37:21.196909 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.196836    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="b5229a93-9882-4b81-8d8b-21bd33a3951f" containerName="kube-rbac-proxy"
May 20 14:37:21.200125 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.200103    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.203179 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.203152    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
May 20 14:37:21.203279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.203226    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
May 20 14:37:21.212421 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.212396    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"]
May 20 14:37:21.331714 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.331690    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.331848 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.331721    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8tg\" (UniqueName: \"kubernetes.io/projected/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kube-api-access-4z8tg\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.331848 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.331742    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d6dcda-c954-441a-b8f0-4ed5382ee1de-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.331848 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.331813    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.332051 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.331924    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.432886 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.432810    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.432979 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.432901    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.432979 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.432923    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8tg\" (UniqueName: \"kubernetes.io/projected/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kube-api-access-4z8tg\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.432979 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.432944    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d6dcda-c954-441a-b8f0-4ed5382ee1de-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.432979 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.432970    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.433395 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.433371    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.433534 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.433518    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.433582 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.433564    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.435431 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.435412    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d6dcda-c954-441a-b8f0-4ed5382ee1de-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.442064 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.442044    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8tg\" (UniqueName: \"kubernetes.io/projected/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kube-api-access-4z8tg\") pod \"isvc-sklearn-runtime-predictor-d76b964b5-jrx4t\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.511224 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.511198    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:21.646160 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:21.646128    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"]
May 20 14:37:21.648815 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:37:21.648788    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d6dcda_c954_441a_b8f0_4ed5382ee1de.slice/crio-2a416dd374e2bad4040f051732025f9bb09c07e2253e0b81b4d4d70e0333eb34 WatchSource:0}: Error finding container 2a416dd374e2bad4040f051732025f9bb09c07e2253e0b81b4d4d70e0333eb34: Status 404 returned error can't find the container with id 2a416dd374e2bad4040f051732025f9bb09c07e2253e0b81b4d4d70e0333eb34
May 20 14:37:22.042572 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:22.042483    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" event={"ID":"25d6dcda-c954-441a-b8f0-4ed5382ee1de","Type":"ContainerStarted","Data":"c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946"}
May 20 14:37:22.042572 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:22.042519    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" event={"ID":"25d6dcda-c954-441a-b8f0-4ed5382ee1de","Type":"ContainerStarted","Data":"2a416dd374e2bad4040f051732025f9bb09c07e2253e0b81b4d4d70e0333eb34"}
May 20 14:37:22.044342 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:22.044316    2557 generic.go:358] "Generic (PLEG): container finished" podID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerID="85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9" exitCode=2
May 20 14:37:22.044461 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:22.044393    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" event={"ID":"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8","Type":"ContainerDied","Data":"85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9"}
May 20 14:37:22.868385 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:22.868346    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
May 20 14:37:23.894821 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:23.894751    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.51:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
May 20 14:37:27.065815 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.065787    2557 generic.go:358] "Generic (PLEG): container finished" podID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerID="c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946" exitCode=0
May 20 14:37:27.066206 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.065875    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" event={"ID":"25d6dcda-c954-441a-b8f0-4ed5382ee1de","Type":"ContainerDied","Data":"c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946"}
May 20 14:37:27.255141 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.255116    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:37:27.273401 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.273377    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-proxy-tls\") pod \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") "
May 20 14:37:27.273513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.273416    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6rm4\" (UniqueName: \"kubernetes.io/projected/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kube-api-access-f6rm4\") pod \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") "
May 20 14:37:27.273513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.273471    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-cabundle-cert\") pod \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") "
May 20 14:37:27.273513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.273509    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kserve-provision-location\") pod \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") "
May 20 14:37:27.273681 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.273550    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\" (UID: \"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8\") "
May 20 14:37:27.273840 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.273813    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" (UID: "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:37:27.273840 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.273811    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" (UID: "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:37:27.274085 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.274061    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" (UID: "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:37:27.275412 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.275387    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kube-api-access-f6rm4" (OuterVolumeSpecName: "kube-api-access-f6rm4") pod "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" (UID: "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8"). InnerVolumeSpecName "kube-api-access-f6rm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:37:27.276203 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.275757    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" (UID: "0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:37:27.374209 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.374177    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:37:27.374209 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.374207    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rm4\" (UniqueName: \"kubernetes.io/projected/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kube-api-access-f6rm4\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:37:27.374209 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.374218    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:37:27.374450 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.374228    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:37:27.374450 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:27.374238    2557 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:37:28.072059 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.072021    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" event={"ID":"25d6dcda-c954-441a-b8f0-4ed5382ee1de","Type":"ContainerStarted","Data":"a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733"}
May 20 14:37:28.072500 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.072075    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" event={"ID":"25d6dcda-c954-441a-b8f0-4ed5382ee1de","Type":"ContainerStarted","Data":"c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861"}
May 20 14:37:28.072500 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.072398    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:28.072645 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.072521    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:28.073872 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.073826    2557 generic.go:358] "Generic (PLEG): container finished" podID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerID="a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc" exitCode=0
May 20 14:37:28.073973 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.073848    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" event={"ID":"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8","Type":"ContainerDied","Data":"a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc"}
May 20 14:37:28.073973 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.073902    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk" event={"ID":"0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8","Type":"ContainerDied","Data":"318393c251746e99ae9733344d763e392714bd59f14c14eb972c29e7f5155439"}
May 20 14:37:28.073973 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.073822    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 20 14:37:28.073973 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.073934    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"
May 20 14:37:28.074188 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.073977    2557 scope.go:117] "RemoveContainer" containerID="85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9"
May 20 14:37:28.084269 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.084251    2557 scope.go:117] "RemoveContainer" containerID="a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc"
May 20 14:37:28.091939 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.091924    2557 scope.go:117] "RemoveContainer" containerID="2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38"
May 20 14:37:28.095402 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.095363    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" podStartSLOduration=7.095352078 podStartE2EDuration="7.095352078s" podCreationTimestamp="2026-05-20 14:37:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:37:28.094011072 +0000 UTC m=+2828.231565292" watchObservedRunningTime="2026-05-20 14:37:28.095352078 +0000 UTC m=+2828.232906296"
May 20 14:37:28.099681 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.099664    2557 scope.go:117] "RemoveContainer" containerID="85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9"
May 20 14:37:28.099944 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:37:28.099919    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9\": container with ID starting with 85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9 not found: ID does not exist" containerID="85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9"
May 20 14:37:28.100028 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.099956    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9"} err="failed to get container status \"85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9\": rpc error: code = NotFound desc = could not find container \"85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9\": container with ID starting with 85913eac80668474ae780c76f7371bdc1f851904b8351bef18fae90171feede9 not found: ID does not exist"
May 20 14:37:28.100028 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.099984    2557 scope.go:117] "RemoveContainer" containerID="a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc"
May 20 14:37:28.100206 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:37:28.100190    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc\": container with ID starting with a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc not found: ID does not exist" containerID="a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc"
May 20 14:37:28.100252 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.100209    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc"} err="failed to get container status \"a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc\": rpc error: code = NotFound desc = could not find container \"a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc\": container with ID starting with a5138a2da54a73721360970dd2459313d90e70cdda0d059953d8fe0149b8f4dc not found: ID does not exist"
May 20 14:37:28.100252 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.100223    2557 scope.go:117] "RemoveContainer" containerID="2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38"
May 20 14:37:28.100474 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:37:28.100456    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38\": container with ID starting with 2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38 not found: ID does not exist" containerID="2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38"
May 20 14:37:28.100530 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.100483    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38"} err="failed to get container status \"2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38\": rpc error: code = NotFound desc = could not find container \"2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38\": container with ID starting with 2fc12c2fc2cda8bedf08cf19eab2256566e48026f3008ce07f7d5d9538fa2a38 not found: ID does not exist"
May 20 14:37:28.111657 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.111635    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"]
May 20 14:37:28.122334 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.122313    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-zbrpk"]
May 20 14:37:28.405978 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:28.405895    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" path="/var/lib/kubelet/pods/0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8/volumes"
May 20 14:37:29.078480 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:29.078440    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 20 14:37:34.083051 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:34.083002    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:34.083729 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:34.083700    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 20 14:37:44.084827 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:44.084792    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:58.274299 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.274256    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-d76b964b5-jrx4t_25d6dcda-c954-441a-b8f0-4ed5382ee1de/kserve-container/0.log"
May 20 14:37:58.461132 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.461097    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"]
May 20 14:37:58.461611 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.461567    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kube-rbac-proxy" containerID="cri-o://a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733" gracePeriod=30
May 20 14:37:58.461743 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.461548    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kserve-container" containerID="cri-o://c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861" gracePeriod=30
May 20 14:37:58.573830 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.573801    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"]
May 20 14:37:58.574246 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.574226    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="storage-initializer"
May 20 14:37:58.574303 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.574249    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="storage-initializer"
May 20 14:37:58.574303 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.574263    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kube-rbac-proxy"
May 20 14:37:58.574303 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.574273    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kube-rbac-proxy"
May 20 14:37:58.574303 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.574289    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kserve-container"
May 20 14:37:58.574303 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.574295    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kserve-container"
May 20 14:37:58.574463 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.574358    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kserve-container"
May 20 14:37:58.574463 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.574368    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="0f1d10dc-4d4c-4f1d-af1a-670e975b4ac8" containerName="kube-rbac-proxy"
May 20 14:37:58.577108 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.577090    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.580643 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.580626    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
May 20 14:37:58.581644 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.581629    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
May 20 14:37:58.590265 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.590240    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"]
May 20 14:37:58.723876 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.723828    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.723876 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.723874    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269c581d-e488-4266-9762-4390fb8b1c8c-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.724013 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.723895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269c581d-e488-4266-9762-4390fb8b1c8c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.724051 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.724005    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfm7\" (UniqueName: \"kubernetes.io/projected/269c581d-e488-4266-9762-4390fb8b1c8c-kube-api-access-xjfm7\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.724051 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.724033    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.824653 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.824594    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfm7\" (UniqueName: \"kubernetes.io/projected/269c581d-e488-4266-9762-4390fb8b1c8c-kube-api-access-xjfm7\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.824653 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.824625    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.824789 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.824674    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.824789 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.824705    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269c581d-e488-4266-9762-4390fb8b1c8c-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.824789 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.824734    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269c581d-e488-4266-9762-4390fb8b1c8c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.824947 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:37:58.824870    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
May 20 14:37:58.824947 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:37:58.824926    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/269c581d-e488-4266-9762-4390fb8b1c8c-proxy-tls podName:269c581d-e488-4266-9762-4390fb8b1c8c nodeName:}" failed. No retries permitted until 2026-05-20 14:37:59.324907854 +0000 UTC m=+2859.462462051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/269c581d-e488-4266-9762-4390fb8b1c8c-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" (UID: "269c581d-e488-4266-9762-4390fb8b1c8c") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
May 20 14:37:58.825175 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.825153    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269c581d-e488-4266-9762-4390fb8b1c8c-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.825379 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.825365    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.825475 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.825456    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:58.844901 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:58.844850    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfm7\" (UniqueName: \"kubernetes.io/projected/269c581d-e488-4266-9762-4390fb8b1c8c-kube-api-access-xjfm7\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:59.078774 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.078678    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
May 20 14:37:59.179140 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.179104    2557 generic.go:358] "Generic (PLEG): container finished" podID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerID="a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733" exitCode=2
May 20 14:37:59.179321 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.179183    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" event={"ID":"25d6dcda-c954-441a-b8f0-4ed5382ee1de","Type":"ContainerDied","Data":"a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733"}
May 20 14:37:59.329107 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.329083    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269c581d-e488-4266-9762-4390fb8b1c8c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:59.332180 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.332152    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269c581d-e488-4266-9762-4390fb8b1c8c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:59.406639 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.406619    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:37:59.487326 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.487297    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:37:59.530396 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.530365    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kserve-provision-location\") pod \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") "
May 20 14:37:59.530535 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.530405    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") "
May 20 14:37:59.530535 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.530483    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-cabundle-cert\") pod \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") "
May 20 14:37:59.530535 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.530514    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d6dcda-c954-441a-b8f0-4ed5382ee1de-proxy-tls\") pod \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") "
May 20 14:37:59.530704 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.530589    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z8tg\" (UniqueName: \"kubernetes.io/projected/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kube-api-access-4z8tg\") pod \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\" (UID: \"25d6dcda-c954-441a-b8f0-4ed5382ee1de\") "
May 20 14:37:59.530896 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.530865    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "25d6dcda-c954-441a-b8f0-4ed5382ee1de" (UID: "25d6dcda-c954-441a-b8f0-4ed5382ee1de"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:37:59.530968 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.530893    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "25d6dcda-c954-441a-b8f0-4ed5382ee1de" (UID: "25d6dcda-c954-441a-b8f0-4ed5382ee1de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:37:59.532842 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.532806    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d6dcda-c954-441a-b8f0-4ed5382ee1de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25d6dcda-c954-441a-b8f0-4ed5382ee1de" (UID: "25d6dcda-c954-441a-b8f0-4ed5382ee1de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:37:59.533084 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.533049    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kube-api-access-4z8tg" (OuterVolumeSpecName: "kube-api-access-4z8tg") pod "25d6dcda-c954-441a-b8f0-4ed5382ee1de" (UID: "25d6dcda-c954-441a-b8f0-4ed5382ee1de"). InnerVolumeSpecName "kube-api-access-4z8tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:37:59.558405 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.558379    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25d6dcda-c954-441a-b8f0-4ed5382ee1de" (UID: "25d6dcda-c954-441a-b8f0-4ed5382ee1de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:37:59.619727 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.619703    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"]
May 20 14:37:59.622115 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:37:59.622085    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269c581d_e488_4266_9762_4390fb8b1c8c.slice/crio-bfd2d5f4135660156afa2889c2be162fe137d60d6b6cc478c50b8514750b817e WatchSource:0}: Error finding container bfd2d5f4135660156afa2889c2be162fe137d60d6b6cc478c50b8514750b817e: Status 404 returned error can't find the container with id bfd2d5f4135660156afa2889c2be162fe137d60d6b6cc478c50b8514750b817e
May 20 14:37:59.631664 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.631642    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z8tg\" (UniqueName: \"kubernetes.io/projected/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kube-api-access-4z8tg\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:37:59.631751 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.631666    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25d6dcda-c954-441a-b8f0-4ed5382ee1de-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:37:59.631751 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.631681    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:37:59.631751 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.631696    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25d6dcda-c954-441a-b8f0-4ed5382ee1de-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:37:59.631751 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:37:59.631709    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25d6dcda-c954-441a-b8f0-4ed5382ee1de-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:38:00.189959 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.189922    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" event={"ID":"269c581d-e488-4266-9762-4390fb8b1c8c","Type":"ContainerStarted","Data":"5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac"}
May 20 14:38:00.189959 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.189960    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" event={"ID":"269c581d-e488-4266-9762-4390fb8b1c8c","Type":"ContainerStarted","Data":"bfd2d5f4135660156afa2889c2be162fe137d60d6b6cc478c50b8514750b817e"}
May 20 14:38:00.191610 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.191579    2557 generic.go:358] "Generic (PLEG): container finished" podID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerID="c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861" exitCode=0
May 20 14:38:00.191737 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.191655    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" event={"ID":"25d6dcda-c954-441a-b8f0-4ed5382ee1de","Type":"ContainerDied","Data":"c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861"}
May 20 14:38:00.191737 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.191687    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t" event={"ID":"25d6dcda-c954-441a-b8f0-4ed5382ee1de","Type":"ContainerDied","Data":"2a416dd374e2bad4040f051732025f9bb09c07e2253e0b81b4d4d70e0333eb34"}
May 20 14:38:00.191737 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.191703    2557 scope.go:117] "RemoveContainer" containerID="a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733"
May 20 14:38:00.191921 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.191663    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"
May 20 14:38:00.200012 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.199995    2557 scope.go:117] "RemoveContainer" containerID="c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861"
May 20 14:38:00.207108 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.207091    2557 scope.go:117] "RemoveContainer" containerID="c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946"
May 20 14:38:00.215461 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.215442    2557 scope.go:117] "RemoveContainer" containerID="a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733"
May 20 14:38:00.215709 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:38:00.215690    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733\": container with ID starting with a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733 not found: ID does not exist" containerID="a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733"
May 20 14:38:00.215760 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.215717    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733"} err="failed to get container status \"a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733\": rpc error: code = NotFound desc = could not find container \"a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733\": container with ID starting with a8090718c1fe98159113e77bb7e75c23b7e22cd6f04ef9968cf13904e774f733 not found: ID does not exist"
May 20 14:38:00.215760 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.215734    2557 scope.go:117] "RemoveContainer" containerID="c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861"
May 20 14:38:00.216165 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:38:00.216131    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861\": container with ID starting with c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861 not found: ID does not exist" containerID="c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861"
May 20 14:38:00.216230 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.216165    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861"} err="failed to get container status \"c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861\": rpc error: code = NotFound desc = could not find container \"c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861\": container with ID starting with c2777bef0207f1801e6cae1e40601cb60dd753fbdd19cec00d47072d7d08c861 not found: ID does not exist"
May 20 14:38:00.216230 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.216181    2557 scope.go:117] "RemoveContainer" containerID="c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946"
May 20 14:38:00.216420 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:38:00.216406    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946\": container with ID starting with c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946 not found: ID does not exist" containerID="c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946"
May 20 14:38:00.216460 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.216425    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946"} err="failed to get container status \"c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946\": rpc error: code = NotFound desc = could not find container \"c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946\": container with ID starting with c35998cd56a7ae525cec07c8a9386459f34d1fd5a2f1db027e869f032affc946 not found: ID does not exist"
May 20 14:38:00.228658 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.228633    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"]
May 20 14:38:00.233178 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.233158    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d76b964b5-jrx4t"]
May 20 14:38:00.405316 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:00.405291    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" path="/var/lib/kubelet/pods/25d6dcda-c954-441a-b8f0-4ed5382ee1de/volumes"
May 20 14:38:04.214029 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:04.213988    2557 generic.go:358] "Generic (PLEG): container finished" podID="269c581d-e488-4266-9762-4390fb8b1c8c" containerID="5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac" exitCode=0
May 20 14:38:04.214447 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:04.214069    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" event={"ID":"269c581d-e488-4266-9762-4390fb8b1c8c","Type":"ContainerDied","Data":"5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac"}
May 20 14:38:05.219715 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:05.219683    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" event={"ID":"269c581d-e488-4266-9762-4390fb8b1c8c","Type":"ContainerStarted","Data":"59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d"}
May 20 14:38:05.220127 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:05.219723    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" event={"ID":"269c581d-e488-4266-9762-4390fb8b1c8c","Type":"ContainerStarted","Data":"f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308"}
May 20 14:38:05.220127 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:05.220056    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:38:05.220127 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:05.220099    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:38:05.242442 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:05.242395    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" podStartSLOduration=7.242382981 podStartE2EDuration="7.242382981s" podCreationTimestamp="2026-05-20 14:37:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:38:05.241392776 +0000 UTC m=+2865.378947023" watchObservedRunningTime="2026-05-20 14:38:05.242382981 +0000 UTC m=+2865.379937200"
May 20 14:38:11.229845 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:11.229811    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:38:41.288722 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:41.288680    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 20 14:38:51.232645 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:51.232615    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:38:58.602935 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.602902    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"]
May 20 14:38:58.603417 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.603212    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kserve-container" containerID="cri-o://f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308" gracePeriod=30
May 20 14:38:58.603417 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.603239    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kube-rbac-proxy" containerID="cri-o://59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d" gracePeriod=30
May 20 14:38:58.711613 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.711572    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"]
May 20 14:38:58.712034 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.712016    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kserve-container"
May 20 14:38:58.712159 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.712036    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kserve-container"
May 20 14:38:58.712159 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.712073    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kube-rbac-proxy"
May 20 14:38:58.712159 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.712082    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kube-rbac-proxy"
May 20 14:38:58.712159 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.712103    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="storage-initializer"
May 20 14:38:58.712159 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.712111    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="storage-initializer"
May 20 14:38:58.712663 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.712185    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kserve-container"
May 20 14:38:58.712663 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.712203    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="25d6dcda-c954-441a-b8f0-4ed5382ee1de" containerName="kube-rbac-proxy"
May 20 14:38:58.716001 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.715984    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.719289 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.719271    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
May 20 14:38:58.719369 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.719325    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
May 20 14:38:58.729412 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.729388    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"]
May 20 14:38:58.772143 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.772122    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jmq\" (UniqueName: \"kubernetes.io/projected/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kube-api-access-r5jmq\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.772237 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.772162    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.772237 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.772210    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06298e9a-323d-471d-89c9-2d5eb9aa93ab-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.772325 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.772304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.772380 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.772331    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.872691 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.872640    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.872691 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.872667    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.872842 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.872713    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jmq\" (UniqueName: \"kubernetes.io/projected/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kube-api-access-r5jmq\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.872842 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.872733    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.872842 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.872753    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06298e9a-323d-471d-89c9-2d5eb9aa93ab-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.873042 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:38:58.872850    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
May 20 14:38:58.873042 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:38:58.872927    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06298e9a-323d-471d-89c9-2d5eb9aa93ab-proxy-tls podName:06298e9a-323d-471d-89c9-2d5eb9aa93ab nodeName:}" failed. No retries permitted until 2026-05-20 14:38:59.372908712 +0000 UTC m=+2919.510462909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/06298e9a-323d-471d-89c9-2d5eb9aa93ab-proxy-tls") pod "isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" (UID: "06298e9a-323d-471d-89c9-2d5eb9aa93ab") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
May 20 14:38:58.873164 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.873131    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.873430 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.873409    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.873468 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.873453    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:58.881960 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:58.881938    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jmq\" (UniqueName: \"kubernetes.io/projected/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kube-api-access-r5jmq\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:59.376933 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:59.376902    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06298e9a-323d-471d-89c9-2d5eb9aa93ab-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:59.379248 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:59.379221    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06298e9a-323d-471d-89c9-2d5eb9aa93ab-proxy-tls\") pod \"isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:59.401435 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:59.401402    2557 generic.go:358] "Generic (PLEG): container finished" podID="269c581d-e488-4266-9762-4390fb8b1c8c" containerID="59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d" exitCode=2
May 20 14:38:59.401542 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:59.401470    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" event={"ID":"269c581d-e488-4266-9762-4390fb8b1c8c","Type":"ContainerDied","Data":"59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d"}
May 20 14:38:59.630443 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:59.630359    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:38:59.759189 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:38:59.759127    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"]
May 20 14:38:59.761711 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:38:59.761671    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06298e9a_323d_471d_89c9_2d5eb9aa93ab.slice/crio-24888a77a5508a1b7151885284964883b88a266c4a793194fb2c7356c5858610 WatchSource:0}: Error finding container 24888a77a5508a1b7151885284964883b88a266c4a793194fb2c7356c5858610: Status 404 returned error can't find the container with id 24888a77a5508a1b7151885284964883b88a266c4a793194fb2c7356c5858610
May 20 14:39:00.406234 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:00.406200    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" event={"ID":"06298e9a-323d-471d-89c9-2d5eb9aa93ab","Type":"ContainerStarted","Data":"2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856"}
May 20 14:39:00.406234 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:00.406236    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" event={"ID":"06298e9a-323d-471d-89c9-2d5eb9aa93ab","Type":"ContainerStarted","Data":"24888a77a5508a1b7151885284964883b88a266c4a793194fb2c7356c5858610"}
May 20 14:39:01.224558 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:01.224509    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
May 20 14:39:04.418698 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:04.418659    2557 generic.go:358] "Generic (PLEG): container finished" podID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerID="2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856" exitCode=0
May 20 14:39:04.418698 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:04.418694    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" event={"ID":"06298e9a-323d-471d-89c9-2d5eb9aa93ab","Type":"ContainerDied","Data":"2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856"}
May 20 14:39:05.043549 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.043526    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:39:05.127979 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.127953    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-cabundle-cert\") pod \"269c581d-e488-4266-9762-4390fb8b1c8c\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") "
May 20 14:39:05.128090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.127986    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269c581d-e488-4266-9762-4390fb8b1c8c-proxy-tls\") pod \"269c581d-e488-4266-9762-4390fb8b1c8c\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") "
May 20 14:39:05.128090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.128018    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269c581d-e488-4266-9762-4390fb8b1c8c-kserve-provision-location\") pod \"269c581d-e488-4266-9762-4390fb8b1c8c\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") "
May 20 14:39:05.128090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.128053    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"269c581d-e488-4266-9762-4390fb8b1c8c\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") "
May 20 14:39:05.128090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.128083    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjfm7\" (UniqueName: \"kubernetes.io/projected/269c581d-e488-4266-9762-4390fb8b1c8c-kube-api-access-xjfm7\") pod \"269c581d-e488-4266-9762-4390fb8b1c8c\" (UID: \"269c581d-e488-4266-9762-4390fb8b1c8c\") "
May 20 14:39:05.128399 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.128372    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "269c581d-e488-4266-9762-4390fb8b1c8c" (UID: "269c581d-e488-4266-9762-4390fb8b1c8c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:39:05.128475 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.128394    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269c581d-e488-4266-9762-4390fb8b1c8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "269c581d-e488-4266-9762-4390fb8b1c8c" (UID: "269c581d-e488-4266-9762-4390fb8b1c8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:39:05.128475 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.128460    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "269c581d-e488-4266-9762-4390fb8b1c8c" (UID: "269c581d-e488-4266-9762-4390fb8b1c8c"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:39:05.130049 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.130027    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269c581d-e488-4266-9762-4390fb8b1c8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "269c581d-e488-4266-9762-4390fb8b1c8c" (UID: "269c581d-e488-4266-9762-4390fb8b1c8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:39:05.130144 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.130102    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269c581d-e488-4266-9762-4390fb8b1c8c-kube-api-access-xjfm7" (OuterVolumeSpecName: "kube-api-access-xjfm7") pod "269c581d-e488-4266-9762-4390fb8b1c8c" (UID: "269c581d-e488-4266-9762-4390fb8b1c8c"). InnerVolumeSpecName "kube-api-access-xjfm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:39:05.229451 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.229393    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:39:05.229451 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.229418    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjfm7\" (UniqueName: \"kubernetes.io/projected/269c581d-e488-4266-9762-4390fb8b1c8c-kube-api-access-xjfm7\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:39:05.229451 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.229432    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/269c581d-e488-4266-9762-4390fb8b1c8c-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:39:05.229451 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.229446    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/269c581d-e488-4266-9762-4390fb8b1c8c-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:39:05.229681 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.229460    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/269c581d-e488-4266-9762-4390fb8b1c8c-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:39:05.423587 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.423557    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" event={"ID":"06298e9a-323d-471d-89c9-2d5eb9aa93ab","Type":"ContainerStarted","Data":"679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014"}
May 20 14:39:05.423943 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.423594    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" event={"ID":"06298e9a-323d-471d-89c9-2d5eb9aa93ab","Type":"ContainerStarted","Data":"35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59"}
May 20 14:39:05.424012 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.423966    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:39:05.424012 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.423995    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:39:05.425346 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.425318    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
May 20 14:39:05.425346 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.425341    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" event={"ID":"269c581d-e488-4266-9762-4390fb8b1c8c","Type":"ContainerDied","Data":"f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308"}
May 20 14:39:05.425530 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.425374    2557 scope.go:117] "RemoveContainer" containerID="59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d"
May 20 14:39:05.425530 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.425323    2557 generic.go:358] "Generic (PLEG): container finished" podID="269c581d-e488-4266-9762-4390fb8b1c8c" containerID="f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308" exitCode=0
May 20 14:39:05.425530 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.425421    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"
May 20 14:39:05.425530 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.425507    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb" event={"ID":"269c581d-e488-4266-9762-4390fb8b1c8c","Type":"ContainerDied","Data":"bfd2d5f4135660156afa2889c2be162fe137d60d6b6cc478c50b8514750b817e"}
May 20 14:39:05.433790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.433776    2557 scope.go:117] "RemoveContainer" containerID="f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308"
May 20 14:39:05.440812 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.440789    2557 scope.go:117] "RemoveContainer" containerID="5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac"
May 20 14:39:05.448006 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.447806    2557 scope.go:117] "RemoveContainer" containerID="59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d"
May 20 14:39:05.448175 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:39:05.448122    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d\": container with ID starting with 59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d not found: ID does not exist" containerID="59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d"
May 20 14:39:05.448259 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.448163    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d"} err="failed to get container status \"59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d\": rpc error: code = NotFound desc = could not find container \"59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d\": container with ID starting with 59e64db368cac8aabfa069f388494464dca17c438ebcdfb1c49d00170774df8d not found: ID does not exist"
May 20 14:39:05.448259 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.448188    2557 scope.go:117] "RemoveContainer" containerID="f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308"
May 20 14:39:05.448504 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:39:05.448485    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308\": container with ID starting with f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308 not found: ID does not exist" containerID="f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308"
May 20 14:39:05.448586 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.448511    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308"} err="failed to get container status \"f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308\": rpc error: code = NotFound desc = could not find container \"f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308\": container with ID starting with f83b1ae67f843b2509dc4d8977117b1dcd9f1a17ea36aee9c2691950439e3308 not found: ID does not exist"
May 20 14:39:05.448586 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.448532    2557 scope.go:117] "RemoveContainer" containerID="5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac"
May 20 14:39:05.448771 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:39:05.448756    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac\": container with ID starting with 5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac not found: ID does not exist" containerID="5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac"
May 20 14:39:05.448833 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.448803    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac"} err="failed to get container status \"5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac\": rpc error: code = NotFound desc = could not find container \"5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac\": container with ID starting with 5ce6189b4b0ee79cadc5e77df3f95085ea896a758e1012e78ac6258e3ca004ac not found: ID does not exist"
May 20 14:39:05.449813 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.449778    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podStartSLOduration=7.449767711 podStartE2EDuration="7.449767711s" podCreationTimestamp="2026-05-20 14:38:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:39:05.446449616 +0000 UTC m=+2925.584003835" watchObservedRunningTime="2026-05-20 14:39:05.449767711 +0000 UTC m=+2925.587321930"
May 20 14:39:05.461765 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.461737    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"]
May 20 14:39:05.466544 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:05.466524    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-8rpvb"]
May 20 14:39:06.405010 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:06.404977    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" path="/var/lib/kubelet/pods/269c581d-e488-4266-9762-4390fb8b1c8c/volumes"
May 20 14:39:06.429524 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:06.429495    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
May 20 14:39:11.434224 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:11.434199    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:39:11.434649 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:11.434625    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
May 20 14:39:21.434958 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:21.434919    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
May 20 14:39:31.435603 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:31.435520    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
May 20 14:39:41.435036 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:41.434993    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
May 20 14:39:51.434755 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:39:51.434715    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
May 20 14:40:01.435301 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:01.435237    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
May 20 14:40:11.435761 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:11.435730    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:40:19.060571 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.060536    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"]
May 20 14:40:19.061067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.061048    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="storage-initializer"
May 20 14:40:19.061138 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.061070    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="storage-initializer"
May 20 14:40:19.061138 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.061117    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kserve-container"
May 20 14:40:19.061138 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.061127    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kserve-container"
May 20 14:40:19.061138 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.061137    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kube-rbac-proxy"
May 20 14:40:19.061287 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.061146    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kube-rbac-proxy"
May 20 14:40:19.061287 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.061223    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kube-rbac-proxy"
May 20 14:40:19.061287 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.061233    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="269c581d-e488-4266-9762-4390fb8b1c8c" containerName="kserve-container"
May 20 14:40:19.064990 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.064963    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.070656 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.070619    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
May 20 14:40:19.070787 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.070731    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
May 20 14:40:19.086011 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.085981    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"]
May 20 14:40:19.086423 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.086369    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" containerID="cri-o://35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59" gracePeriod=30
May 20 14:40:19.087069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.086487    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kube-rbac-proxy" containerID="cri-o://679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014" gracePeriod=30
May 20 14:40:19.088550 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.088527    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"]
May 20 14:40:19.159006 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.158967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.159164 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.159010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9v7l\" (UniqueName: \"kubernetes.io/projected/2b4090dd-de28-4e27-a00e-b038253f60c7-kube-api-access-v9v7l\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.159164 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.159103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b4090dd-de28-4e27-a00e-b038253f60c7-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.159253 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.159171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.159253 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.159211    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b4090dd-de28-4e27-a00e-b038253f60c7-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.260067 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.260035    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.260206 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.260081    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b4090dd-de28-4e27-a00e-b038253f60c7-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.260206 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.260100    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.260312 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.260216    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9v7l\" (UniqueName: \"kubernetes.io/projected/2b4090dd-de28-4e27-a00e-b038253f60c7-kube-api-access-v9v7l\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.260312 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.260283    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b4090dd-de28-4e27-a00e-b038253f60c7-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.260560 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.260530    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b4090dd-de28-4e27-a00e-b038253f60c7-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.260799 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.260769    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.260866 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.260775    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.262523 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.262504    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b4090dd-de28-4e27-a00e-b038253f60c7-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.276352 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.276322    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9v7l\" (UniqueName: \"kubernetes.io/projected/2b4090dd-de28-4e27-a00e-b038253f60c7-kube-api-access-v9v7l\") pod \"isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.374921 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.374884    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:19.509948 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.509923    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"]
May 20 14:40:19.511849 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:40:19.511821    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4090dd_de28_4e27_a00e_b038253f60c7.slice/crio-521579c55f9331efc97554fba6bacd61f90759305e63d927f83e62d21821e3db WatchSource:0}: Error finding container 521579c55f9331efc97554fba6bacd61f90759305e63d927f83e62d21821e3db: Status 404 returned error can't find the container with id 521579c55f9331efc97554fba6bacd61f90759305e63d927f83e62d21821e3db
May 20 14:40:19.668130 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.668031    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" event={"ID":"2b4090dd-de28-4e27-a00e-b038253f60c7","Type":"ContainerStarted","Data":"b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca"}
May 20 14:40:19.668130 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.668073    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" event={"ID":"2b4090dd-de28-4e27-a00e-b038253f60c7","Type":"ContainerStarted","Data":"521579c55f9331efc97554fba6bacd61f90759305e63d927f83e62d21821e3db"}
May 20 14:40:19.670026 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.670004    2557 generic.go:358] "Generic (PLEG): container finished" podID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerID="679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014" exitCode=2
May 20 14:40:19.670151 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:19.670030    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" event={"ID":"06298e9a-323d-471d-89c9-2d5eb9aa93ab","Type":"ContainerDied","Data":"679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014"}
May 20 14:40:20.556463 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:20.556438    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:40:20.560681 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:20.560645    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:40:20.562393 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:20.562367    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:40:20.569682 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:20.569658    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:40:21.430166 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:21.430124    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
May 20 14:40:21.434616 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:21.434591    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
May 20 14:40:22.932309 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.932285    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:40:22.990848 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.990789    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") "
May 20 14:40:22.990848 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.990831    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-cabundle-cert\") pod \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") "
May 20 14:40:22.991005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.990889    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06298e9a-323d-471d-89c9-2d5eb9aa93ab-proxy-tls\") pod \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") "
May 20 14:40:22.991005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.990914    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5jmq\" (UniqueName: \"kubernetes.io/projected/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kube-api-access-r5jmq\") pod \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") "
May 20 14:40:22.991005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.990969    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kserve-provision-location\") pod \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\" (UID: \"06298e9a-323d-471d-89c9-2d5eb9aa93ab\") "
May 20 14:40:22.991216 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.991186    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "06298e9a-323d-471d-89c9-2d5eb9aa93ab" (UID: "06298e9a-323d-471d-89c9-2d5eb9aa93ab"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:40:22.991311 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.991273    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06298e9a-323d-471d-89c9-2d5eb9aa93ab" (UID: "06298e9a-323d-471d-89c9-2d5eb9aa93ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:40:22.991378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.991335    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06298e9a-323d-471d-89c9-2d5eb9aa93ab" (UID: "06298e9a-323d-471d-89c9-2d5eb9aa93ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:40:22.992868 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.992838    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06298e9a-323d-471d-89c9-2d5eb9aa93ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06298e9a-323d-471d-89c9-2d5eb9aa93ab" (UID: "06298e9a-323d-471d-89c9-2d5eb9aa93ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:40:22.992962 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:22.992938    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kube-api-access-r5jmq" (OuterVolumeSpecName: "kube-api-access-r5jmq") pod "06298e9a-323d-471d-89c9-2d5eb9aa93ab" (UID: "06298e9a-323d-471d-89c9-2d5eb9aa93ab"). InnerVolumeSpecName "kube-api-access-r5jmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:40:23.092495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.092470    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06298e9a-323d-471d-89c9-2d5eb9aa93ab-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:40:23.092495 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.092492    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5jmq\" (UniqueName: \"kubernetes.io/projected/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kube-api-access-r5jmq\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:40:23.092620 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.092502    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06298e9a-323d-471d-89c9-2d5eb9aa93ab-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:40:23.092620 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.092514    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:40:23.092620 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.092523    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06298e9a-323d-471d-89c9-2d5eb9aa93ab-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:40:23.684556 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.684472    2557 generic.go:358] "Generic (PLEG): container finished" podID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerID="b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca" exitCode=0
May 20 14:40:23.684556 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.684544    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" event={"ID":"2b4090dd-de28-4e27-a00e-b038253f60c7","Type":"ContainerDied","Data":"b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca"}
May 20 14:40:23.686355 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.686329    2557 generic.go:358] "Generic (PLEG): container finished" podID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerID="35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59" exitCode=0
May 20 14:40:23.686459 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.686365    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" event={"ID":"06298e9a-323d-471d-89c9-2d5eb9aa93ab","Type":"ContainerDied","Data":"35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59"}
May 20 14:40:23.686459 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.686383    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj" event={"ID":"06298e9a-323d-471d-89c9-2d5eb9aa93ab","Type":"ContainerDied","Data":"24888a77a5508a1b7151885284964883b88a266c4a793194fb2c7356c5858610"}
May 20 14:40:23.686459 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.686398    2557 scope.go:117] "RemoveContainer" containerID="679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014"
May 20 14:40:23.686459 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.686407    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"
May 20 14:40:23.694965 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.694948    2557 scope.go:117] "RemoveContainer" containerID="35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59"
May 20 14:40:23.715687 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.715650    2557 scope.go:117] "RemoveContainer" containerID="2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856"
May 20 14:40:23.728236 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.728179    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"]
May 20 14:40:23.733085 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.733066    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-5b5d98bd7f-stnsj"]
May 20 14:40:23.735110 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.735092    2557 scope.go:117] "RemoveContainer" containerID="679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014"
May 20 14:40:23.735814 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:40:23.735719    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014\": container with ID starting with 679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014 not found: ID does not exist" containerID="679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014"
May 20 14:40:23.735814 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.735773    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014"} err="failed to get container status \"679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014\": rpc error: code = NotFound desc = could not find container \"679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014\": container with ID starting with 679a4e8e6ebe7091ec868ce97900eb94b61b6211fe529c2916c76d9e69725014 not found: ID does not exist"
May 20 14:40:23.736047 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.735797    2557 scope.go:117] "RemoveContainer" containerID="35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59"
May 20 14:40:23.736560 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:40:23.736529    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59\": container with ID starting with 35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59 not found: ID does not exist" containerID="35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59"
May 20 14:40:23.736661 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.736565    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59"} err="failed to get container status \"35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59\": rpc error: code = NotFound desc = could not find container \"35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59\": container with ID starting with 35042df45148513fc0aeeaec0587ed8ff8b9a9cb9b04f8b3e473ea525a60ba59 not found: ID does not exist"
May 20 14:40:23.736661 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.736589    2557 scope.go:117] "RemoveContainer" containerID="2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856"
May 20 14:40:23.737054 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:40:23.737031    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856\": container with ID starting with 2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856 not found: ID does not exist" containerID="2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856"
May 20 14:40:23.737127 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:23.737062    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856"} err="failed to get container status \"2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856\": rpc error: code = NotFound desc = could not find container \"2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856\": container with ID starting with 2c38315ad41521344d932c63eb657770c1b7d75d42b08764027bdb9230171856 not found: ID does not exist"
May 20 14:40:24.406282 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:24.406251    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" path="/var/lib/kubelet/pods/06298e9a-323d-471d-89c9-2d5eb9aa93ab/volumes"
May 20 14:40:24.692199 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:24.692119    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" event={"ID":"2b4090dd-de28-4e27-a00e-b038253f60c7","Type":"ContainerStarted","Data":"91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9"}
May 20 14:40:24.692199 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:24.692164    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" event={"ID":"2b4090dd-de28-4e27-a00e-b038253f60c7","Type":"ContainerStarted","Data":"e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76"}
May 20 14:40:24.692411 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:24.692396    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:24.723374 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:24.723326    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podStartSLOduration=5.723314873 podStartE2EDuration="5.723314873s" podCreationTimestamp="2026-05-20 14:40:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:40:24.721217008 +0000 UTC m=+3004.858771217" watchObservedRunningTime="2026-05-20 14:40:24.723314873 +0000 UTC m=+3004.860869093"
May 20 14:40:25.695593 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:25.695555    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:25.696943 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:25.696918    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 20 14:40:26.698563 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:26.698514    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 20 14:40:31.703344 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:31.703318    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:40:31.703800 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:31.703773    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 20 14:40:41.704715 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:41.704673    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 20 14:40:51.704258 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:40:51.704212    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 20 14:41:01.704398 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:01.704320    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 20 14:41:11.704574 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:11.704531    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 20 14:41:21.704662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:21.704620    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 20 14:41:31.704967 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:31.704940    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:41:39.164876 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.161705    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"]
May 20 14:41:39.164876 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.162294    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" containerID="cri-o://e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76" gracePeriod=30
May 20 14:41:39.164876 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.162576    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kube-rbac-proxy" containerID="cri-o://91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9" gracePeriod=30
May 20 14:41:39.285393 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.285364    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"]
May 20 14:41:39.285748 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.285735    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="storage-initializer"
May 20 14:41:39.285805 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.285749    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="storage-initializer"
May 20 14:41:39.285805 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.285766    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kube-rbac-proxy"
May 20 14:41:39.285805 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.285771    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kube-rbac-proxy"
May 20 14:41:39.285805 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.285779    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container"
May 20 14:41:39.285805 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.285785    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container"
May 20 14:41:39.285996 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.285834    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kserve-container"
May 20 14:41:39.285996 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.285844    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="06298e9a-323d-471d-89c9-2d5eb9aa93ab" containerName="kube-rbac-proxy"
May 20 14:41:39.288866 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.288839    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.293193 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.293165    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
May 20 14:41:39.293308 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.293296    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
May 20 14:41:39.306736 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.306713    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"]
May 20 14:41:39.343932 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.343906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc8140-5ff0-4993-9128-048640d3b0a5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.344037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.343943    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc8140-5ff0-4993-9128-048640d3b0a5-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.344037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.343962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvgk\" (UniqueName: \"kubernetes.io/projected/51fc8140-5ff0-4993-9128-048640d3b0a5-kube-api-access-hxvgk\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.344037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.343986    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.344183 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.344053    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-cabundle-cert\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.445006 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.444934    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc8140-5ff0-4993-9128-048640d3b0a5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.445006 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.444994    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc8140-5ff0-4993-9128-048640d3b0a5-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.445165 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.445026    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvgk\" (UniqueName: \"kubernetes.io/projected/51fc8140-5ff0-4993-9128-048640d3b0a5-kube-api-access-hxvgk\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.445165 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.445053    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.445165 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.445086    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-cabundle-cert\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.445165 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:41:39.445156    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
May 20 14:41:39.445344 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:41:39.445226    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51fc8140-5ff0-4993-9128-048640d3b0a5-proxy-tls podName:51fc8140-5ff0-4993-9128-048640d3b0a5 nodeName:}" failed. No retries permitted until 2026-05-20 14:41:39.945206386 +0000 UTC m=+3080.082760585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/51fc8140-5ff0-4993-9128-048640d3b0a5-proxy-tls") pod "isvc-tensorflow-predictor-6756f669d7-x2hg5" (UID: "51fc8140-5ff0-4993-9128-048640d3b0a5") : secret "isvc-tensorflow-predictor-serving-cert" not found
May 20 14:41:39.445344 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.445326    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc8140-5ff0-4993-9128-048640d3b0a5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.445764 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.445739    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.445818 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.445804    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-cabundle-cert\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.457202 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.457183    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvgk\" (UniqueName: \"kubernetes.io/projected/51fc8140-5ff0-4993-9128-048640d3b0a5-kube-api-access-hxvgk\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.948129 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.948097    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc8140-5ff0-4993-9128-048640d3b0a5-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.950586 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.950564    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc8140-5ff0-4993-9128-048640d3b0a5-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-x2hg5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:39.951885 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.951843    2557 generic.go:358] "Generic (PLEG): container finished" podID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerID="91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9" exitCode=2
May 20 14:41:39.951968 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:39.951887    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" event={"ID":"2b4090dd-de28-4e27-a00e-b038253f60c7","Type":"ContainerDied","Data":"91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9"}
May 20 14:41:40.199174 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:40.199112    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:40.339709 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:40.339683    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"]
May 20 14:41:40.342411 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:41:40.342382    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fc8140_5ff0_4993_9128_048640d3b0a5.slice/crio-807c52a92d620dae62f070ff5ae7845ce60d743c42a997bca3f9669f040b1c59 WatchSource:0}: Error finding container 807c52a92d620dae62f070ff5ae7845ce60d743c42a997bca3f9669f040b1c59: Status 404 returned error can't find the container with id 807c52a92d620dae62f070ff5ae7845ce60d743c42a997bca3f9669f040b1c59
May 20 14:41:40.344247 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:40.344227    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:41:40.956842 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:40.956808    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" event={"ID":"51fc8140-5ff0-4993-9128-048640d3b0a5","Type":"ContainerStarted","Data":"2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51"}
May 20 14:41:40.956842 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:40.956847    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" event={"ID":"51fc8140-5ff0-4993-9128-048640d3b0a5","Type":"ContainerStarted","Data":"807c52a92d620dae62f070ff5ae7845ce60d743c42a997bca3f9669f040b1c59"}
May 20 14:41:41.699528 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:41.699482    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
May 20 14:41:41.703902 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:41.703875    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 20 14:41:42.922780 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.922757    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:41:42.964904 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.964834    2557 generic.go:358] "Generic (PLEG): container finished" podID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerID="e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76" exitCode=0
May 20 14:41:42.965014 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.964891    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" event={"ID":"2b4090dd-de28-4e27-a00e-b038253f60c7","Type":"ContainerDied","Data":"e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76"}
May 20 14:41:42.965014 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.964932    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9" event={"ID":"2b4090dd-de28-4e27-a00e-b038253f60c7","Type":"ContainerDied","Data":"521579c55f9331efc97554fba6bacd61f90759305e63d927f83e62d21821e3db"}
May 20 14:41:42.965014 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.964945    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"
May 20 14:41:42.965014 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.964953    2557 scope.go:117] "RemoveContainer" containerID="91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9"
May 20 14:41:42.972898 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.972848    2557 scope.go:117] "RemoveContainer" containerID="e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76"
May 20 14:41:42.973016 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.973002    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b4090dd-de28-4e27-a00e-b038253f60c7-proxy-tls\") pod \"2b4090dd-de28-4e27-a00e-b038253f60c7\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") "
May 20 14:41:42.973070 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.973041    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"2b4090dd-de28-4e27-a00e-b038253f60c7\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") "
May 20 14:41:42.973201 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.973182    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-cabundle-cert\") pod \"2b4090dd-de28-4e27-a00e-b038253f60c7\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") "
May 20 14:41:42.973314 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.973232    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9v7l\" (UniqueName: \"kubernetes.io/projected/2b4090dd-de28-4e27-a00e-b038253f60c7-kube-api-access-v9v7l\") pod \"2b4090dd-de28-4e27-a00e-b038253f60c7\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") "
May 20 14:41:42.973314 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.973285    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b4090dd-de28-4e27-a00e-b038253f60c7-kserve-provision-location\") pod \"2b4090dd-de28-4e27-a00e-b038253f60c7\" (UID: \"2b4090dd-de28-4e27-a00e-b038253f60c7\") "
May 20 14:41:42.973449 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.973424    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "2b4090dd-de28-4e27-a00e-b038253f60c7" (UID: "2b4090dd-de28-4e27-a00e-b038253f60c7"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:41:42.973585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.973563    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:41:42.973739 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.973650    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b4090dd-de28-4e27-a00e-b038253f60c7" (UID: "2b4090dd-de28-4e27-a00e-b038253f60c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:41:42.973816 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.973776    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4090dd-de28-4e27-a00e-b038253f60c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b4090dd-de28-4e27-a00e-b038253f60c7" (UID: "2b4090dd-de28-4e27-a00e-b038253f60c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:41:42.975601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.975574    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4090dd-de28-4e27-a00e-b038253f60c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b4090dd-de28-4e27-a00e-b038253f60c7" (UID: "2b4090dd-de28-4e27-a00e-b038253f60c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:41:42.975689 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.975578    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4090dd-de28-4e27-a00e-b038253f60c7-kube-api-access-v9v7l" (OuterVolumeSpecName: "kube-api-access-v9v7l") pod "2b4090dd-de28-4e27-a00e-b038253f60c7" (UID: "2b4090dd-de28-4e27-a00e-b038253f60c7"). InnerVolumeSpecName "kube-api-access-v9v7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:41:42.989025 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.989007    2557 scope.go:117] "RemoveContainer" containerID="b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca"
May 20 14:41:42.995615 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.995600    2557 scope.go:117] "RemoveContainer" containerID="91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9"
May 20 14:41:42.995874 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:41:42.995841    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9\": container with ID starting with 91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9 not found: ID does not exist" containerID="91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9"
May 20 14:41:42.995930 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.995884    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9"} err="failed to get container status \"91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9\": rpc error: code = NotFound desc = could not find container \"91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9\": container with ID starting with 91fc2cd640b1e2cb3085512d3e3aecc9114653eff0985dad26f5f60d63445df9 not found: ID does not exist"
May 20 14:41:42.995930 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.995905    2557 scope.go:117] "RemoveContainer" containerID="e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76"
May 20 14:41:42.996149 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:41:42.996131    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76\": container with ID starting with e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76 not found: ID does not exist" containerID="e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76"
May 20 14:41:42.996195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.996155    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76"} err="failed to get container status \"e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76\": rpc error: code = NotFound desc = could not find container \"e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76\": container with ID starting with e6f55dd9d7f0328dbb0d7cbd73c951d6304d0f89c5d6d23f77eebe2fd4680e76 not found: ID does not exist"
May 20 14:41:42.996195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.996171    2557 scope.go:117] "RemoveContainer" containerID="b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca"
May 20 14:41:42.996378 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:41:42.996363    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca\": container with ID starting with b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca not found: ID does not exist" containerID="b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca"
May 20 14:41:42.996418 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:42.996380    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca"} err="failed to get container status \"b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca\": rpc error: code = NotFound desc = could not find container \"b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca\": container with ID starting with b840ef26e5550ff342c1c273c530ac3ea48bf215e5f4a434375885b3045194ca not found: ID does not exist"
May 20 14:41:43.074086 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:43.074066    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b4090dd-de28-4e27-a00e-b038253f60c7-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:41:43.074086 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:43.074084    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9v7l\" (UniqueName: \"kubernetes.io/projected/2b4090dd-de28-4e27-a00e-b038253f60c7-kube-api-access-v9v7l\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:41:43.074230 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:43.074107    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b4090dd-de28-4e27-a00e-b038253f60c7-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:41:43.074230 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:43.074117    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b4090dd-de28-4e27-a00e-b038253f60c7-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:41:43.295127 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:43.295100    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"]
May 20 14:41:43.307015 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:43.306985    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7c7d4b6c98-q9mj9"]
May 20 14:41:44.406598 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:44.406529    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" path="/var/lib/kubelet/pods/2b4090dd-de28-4e27-a00e-b038253f60c7/volumes"
May 20 14:41:44.973899 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:44.973870    2557 generic.go:358] "Generic (PLEG): container finished" podID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerID="2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51" exitCode=0
May 20 14:41:44.974016 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:44.973930    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" event={"ID":"51fc8140-5ff0-4993-9128-048640d3b0a5","Type":"ContainerDied","Data":"2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51"}
May 20 14:41:48.991321 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:48.991299    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" event={"ID":"51fc8140-5ff0-4993-9128-048640d3b0a5","Type":"ContainerStarted","Data":"523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7"}
May 20 14:41:49.997983 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:49.997941    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" event={"ID":"51fc8140-5ff0-4993-9128-048640d3b0a5","Type":"ContainerStarted","Data":"b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e"}
May 20 14:41:49.998398 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:49.998126    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:49.998398 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:49.998251    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:49.999371 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:49.999342    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 20 14:41:50.019057 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:50.019016    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podStartSLOduration=7.069032377 podStartE2EDuration="11.019005557s" podCreationTimestamp="2026-05-20 14:41:39 +0000 UTC" firstStartedPulling="2026-05-20 14:41:44.975052825 +0000 UTC m=+3085.112607023" lastFinishedPulling="2026-05-20 14:41:48.925026003 +0000 UTC m=+3089.062580203" observedRunningTime="2026-05-20 14:41:50.018038761 +0000 UTC m=+3090.155592981" watchObservedRunningTime="2026-05-20 14:41:50.019005557 +0000 UTC m=+3090.156559775"
May 20 14:41:51.001039 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:51.000999    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 20 14:41:56.005353 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:56.005326    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:41:56.005951 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:41:56.005915    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 20 14:42:06.007320 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:06.007266    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:42:19.170641 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:19.170606    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"]
May 20 14:42:19.171221 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:19.171058    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kserve-container" containerID="cri-o://523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7" gracePeriod=30
May 20 14:42:19.171221 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:19.171097    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy" containerID="cri-o://b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e" gracePeriod=30
May 20 14:42:20.100485 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:20.100444    2557 generic.go:358] "Generic (PLEG): container finished" podID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerID="b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e" exitCode=2
May 20 14:42:20.100674 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:20.100502    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" event={"ID":"51fc8140-5ff0-4993-9128-048640d3b0a5","Type":"ContainerDied","Data":"b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e"}
May 20 14:42:21.001344 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:21.001301    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
May 20 14:42:26.001465 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:26.001415    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
May 20 14:42:31.001532 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:31.001451    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
May 20 14:42:31.001991 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:31.001590    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:42:36.001928 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:36.001877    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
May 20 14:42:41.002062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:41.002016    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
May 20 14:42:46.001593 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:46.001539    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
May 20 14:42:49.811346 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.811322    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:42:49.875455 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.875431    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc8140-5ff0-4993-9128-048640d3b0a5-proxy-tls\") pod \"51fc8140-5ff0-4993-9128-048640d3b0a5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") "
May 20 14:42:49.875580 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.875493    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-cabundle-cert\") pod \"51fc8140-5ff0-4993-9128-048640d3b0a5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") "
May 20 14:42:49.875692 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.875662    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"51fc8140-5ff0-4993-9128-048640d3b0a5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") "
May 20 14:42:49.875767 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.875717    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxvgk\" (UniqueName: \"kubernetes.io/projected/51fc8140-5ff0-4993-9128-048640d3b0a5-kube-api-access-hxvgk\") pod \"51fc8140-5ff0-4993-9128-048640d3b0a5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") "
May 20 14:42:49.875767 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.875757    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51fc8140-5ff0-4993-9128-048640d3b0a5" (UID: "51fc8140-5ff0-4993-9128-048640d3b0a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:42:49.875899 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.875774    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc8140-5ff0-4993-9128-048640d3b0a5-kserve-provision-location\") pod \"51fc8140-5ff0-4993-9128-048640d3b0a5\" (UID: \"51fc8140-5ff0-4993-9128-048640d3b0a5\") "
May 20 14:42:49.875984 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.875963    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "51fc8140-5ff0-4993-9128-048640d3b0a5" (UID: "51fc8140-5ff0-4993-9128-048640d3b0a5"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:42:49.876122 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.876108    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:42:49.876191 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.876124    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc8140-5ff0-4993-9128-048640d3b0a5-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:42:49.877400 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.877377    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fc8140-5ff0-4993-9128-048640d3b0a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51fc8140-5ff0-4993-9128-048640d3b0a5" (UID: "51fc8140-5ff0-4993-9128-048640d3b0a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:42:49.877895 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.877867    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fc8140-5ff0-4993-9128-048640d3b0a5-kube-api-access-hxvgk" (OuterVolumeSpecName: "kube-api-access-hxvgk") pod "51fc8140-5ff0-4993-9128-048640d3b0a5" (UID: "51fc8140-5ff0-4993-9128-048640d3b0a5"). InnerVolumeSpecName "kube-api-access-hxvgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:42:49.886928 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.886901    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fc8140-5ff0-4993-9128-048640d3b0a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51fc8140-5ff0-4993-9128-048640d3b0a5" (UID: "51fc8140-5ff0-4993-9128-048640d3b0a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:42:49.977414 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.977344    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxvgk\" (UniqueName: \"kubernetes.io/projected/51fc8140-5ff0-4993-9128-048640d3b0a5-kube-api-access-hxvgk\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:42:49.977414 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.977372    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc8140-5ff0-4993-9128-048640d3b0a5-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:42:49.977414 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:49.977383    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc8140-5ff0-4993-9128-048640d3b0a5-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:42:50.206193 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.206154    2557 generic.go:358] "Generic (PLEG): container finished" podID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerID="523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7" exitCode=137
May 20 14:42:50.206356 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.206242    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"
May 20 14:42:50.206356 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.206241    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" event={"ID":"51fc8140-5ff0-4993-9128-048640d3b0a5","Type":"ContainerDied","Data":"523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7"}
May 20 14:42:50.206356 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.206288    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5" event={"ID":"51fc8140-5ff0-4993-9128-048640d3b0a5","Type":"ContainerDied","Data":"807c52a92d620dae62f070ff5ae7845ce60d743c42a997bca3f9669f040b1c59"}
May 20 14:42:50.206356 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.206310    2557 scope.go:117] "RemoveContainer" containerID="b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e"
May 20 14:42:50.215452 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.215432    2557 scope.go:117] "RemoveContainer" containerID="523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7"
May 20 14:42:50.223649 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.223627    2557 scope.go:117] "RemoveContainer" containerID="2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51"
May 20 14:42:50.230072 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.230009    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"]
May 20 14:42:50.231379 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.231364    2557 scope.go:117] "RemoveContainer" containerID="b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e"
May 20 14:42:50.231642 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:42:50.231619    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e\": container with ID starting with b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e not found: ID does not exist" containerID="b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e"
May 20 14:42:50.231700 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.231652    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e"} err="failed to get container status \"b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e\": rpc error: code = NotFound desc = could not find container \"b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e\": container with ID starting with b8c10bbf315c1b3545adf0efa678a3a54510170948c3159c805ba04c59b9748e not found: ID does not exist"
May 20 14:42:50.231700 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.231672    2557 scope.go:117] "RemoveContainer" containerID="523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7"
May 20 14:42:50.231955 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:42:50.231931    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7\": container with ID starting with 523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7 not found: ID does not exist" containerID="523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7"
May 20 14:42:50.232064 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.231958    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7"} err="failed to get container status \"523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7\": rpc error: code = NotFound desc = could not find container \"523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7\": container with ID starting with 523fa67d007da1ae7479239cab09453b016bcef199ae947e1c1e7be1f6f9e4e7 not found: ID does not exist"
May 20 14:42:50.232064 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.231981    2557 scope.go:117] "RemoveContainer" containerID="2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51"
May 20 14:42:50.232379 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:42:50.232327    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51\": container with ID starting with 2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51 not found: ID does not exist" containerID="2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51"
May 20 14:42:50.232474 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.232370    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51"} err="failed to get container status \"2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51\": rpc error: code = NotFound desc = could not find container \"2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51\": container with ID starting with 2a13bfb96fbac3bf266822ac7659f43841db4ea507e60a705f2fa7c6991a1a51 not found: ID does not exist"
May 20 14:42:50.234231 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.234206    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-x2hg5"]
May 20 14:42:50.405801 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:42:50.405769    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" path="/var/lib/kubelet/pods/51fc8140-5ff0-4993-9128-048640d3b0a5/volumes"
May 20 14:45:20.589055 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:45:20.589026    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:45:20.593848 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:45:20.593819    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:45:20.596670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:45:20.596652    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:45:20.601449 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:45:20.601435    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:47:12.722388 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722312    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"]
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722649    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="storage-initializer"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722660    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="storage-initializer"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722668    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722674    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722687    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kserve-container"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722692    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kserve-container"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722698    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722703    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722714    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kube-rbac-proxy"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722720    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kube-rbac-proxy"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722733    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="storage-initializer"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722738    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="storage-initializer"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722805    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kube-rbac-proxy"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722815    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="51fc8140-5ff0-4993-9128-048640d3b0a5" containerName="kserve-container"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722822    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kube-rbac-proxy"
May 20 14:47:12.724947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.722829    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b4090dd-de28-4e27-a00e-b038253f60c7" containerName="kserve-container"
May 20 14:47:12.725814 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.725798    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.728493 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.728471    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
May 20 14:47:12.728632 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.728474    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 14:47:12.728632 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.728621    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2mlds\""
May 20 14:47:12.728874 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.728835    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:47:12.729728 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.729708    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:47:12.729803 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.729742    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:47:12.729803 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.729716    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 20 14:47:12.747393 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.747375    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"]
May 20 14:47:12.893385 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.893353    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.893551 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.893402    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.893551 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.893476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfpb\" (UniqueName: \"kubernetes.io/projected/65fe82b6-52c9-4b82-bedb-c9504247971e-kube-api-access-tbfpb\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.893551 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.893526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fe82b6-52c9-4b82-bedb-c9504247971e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.893684 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.893558    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fe82b6-52c9-4b82-bedb-c9504247971e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.994692 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.994617    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.994692 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.994662    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.994692 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.994683    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfpb\" (UniqueName: \"kubernetes.io/projected/65fe82b6-52c9-4b82-bedb-c9504247971e-kube-api-access-tbfpb\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.994957 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.994709    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fe82b6-52c9-4b82-bedb-c9504247971e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.994957 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.994734    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fe82b6-52c9-4b82-bedb-c9504247971e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.995143 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.995123    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fe82b6-52c9-4b82-bedb-c9504247971e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.995407 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.995386    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.995497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.995386    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:12.997193 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:12.997174    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fe82b6-52c9-4b82-bedb-c9504247971e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:13.007370 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:13.007345    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfpb\" (UniqueName: \"kubernetes.io/projected/65fe82b6-52c9-4b82-bedb-c9504247971e-kube-api-access-tbfpb\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:13.036667 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:13.036644    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:13.165642 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:13.165616    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"]
May 20 14:47:13.168343 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:47:13.168314    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fe82b6_52c9_4b82_bedb_c9504247971e.slice/crio-e85091a2630dc5fc194dbca537b2b879e2d4acae2584c1d8966ba74ba9953fd9 WatchSource:0}: Error finding container e85091a2630dc5fc194dbca537b2b879e2d4acae2584c1d8966ba74ba9953fd9: Status 404 returned error can't find the container with id e85091a2630dc5fc194dbca537b2b879e2d4acae2584c1d8966ba74ba9953fd9
May 20 14:47:13.170184 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:13.170163    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:47:14.056259 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:14.056226    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" event={"ID":"65fe82b6-52c9-4b82-bedb-c9504247971e","Type":"ContainerStarted","Data":"734b2014ac97b5929b4a492b101645a6660dac2b5e98cf992607446b5877f458"}
May 20 14:47:14.056663 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:14.056269    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" event={"ID":"65fe82b6-52c9-4b82-bedb-c9504247971e","Type":"ContainerStarted","Data":"e85091a2630dc5fc194dbca537b2b879e2d4acae2584c1d8966ba74ba9953fd9"}
May 20 14:47:17.067176 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:17.067141    2557 generic.go:358] "Generic (PLEG): container finished" podID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerID="734b2014ac97b5929b4a492b101645a6660dac2b5e98cf992607446b5877f458" exitCode=0
May 20 14:47:17.067512 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:17.067216    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" event={"ID":"65fe82b6-52c9-4b82-bedb-c9504247971e","Type":"ContainerDied","Data":"734b2014ac97b5929b4a492b101645a6660dac2b5e98cf992607446b5877f458"}
May 20 14:47:18.072302 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:18.072268    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" event={"ID":"65fe82b6-52c9-4b82-bedb-c9504247971e","Type":"ContainerStarted","Data":"abafd989331e2bfafa5995808388fcf767163b599d45f289b746a18b21b61269"}
May 20 14:47:18.072658 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:18.072307    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" event={"ID":"65fe82b6-52c9-4b82-bedb-c9504247971e","Type":"ContainerStarted","Data":"015b310dcd669393fd715dfa5c5b5b935af2170196d3a3d134148f5ab12a92ac"}
May 20 14:47:18.072658 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:18.072505    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:18.096340 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:18.096295    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" podStartSLOduration=6.096283378 podStartE2EDuration="6.096283378s" podCreationTimestamp="2026-05-20 14:47:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:47:18.093782551 +0000 UTC m=+3418.231336772" watchObservedRunningTime="2026-05-20 14:47:18.096283378 +0000 UTC m=+3418.233837597"
May 20 14:47:19.076139 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:19.076104    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:25.085380 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:25.085349    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:47:55.089208 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:47:55.089179    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:48:02.788843 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.788807    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"]
May 20 14:48:02.789356 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.789322    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kube-rbac-proxy" containerID="cri-o://abafd989331e2bfafa5995808388fcf767163b599d45f289b746a18b21b61269" gracePeriod=30
May 20 14:48:02.790252 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.789706    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kserve-container" containerID="cri-o://015b310dcd669393fd715dfa5c5b5b935af2170196d3a3d134148f5ab12a92ac" gracePeriod=30
May 20 14:48:02.872213 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.872184    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"]
May 20 14:48:02.875352 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.875335    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:02.878180 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.878161    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
May 20 14:48:02.878274 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.878163    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 20 14:48:02.886656 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.886632    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"]
May 20 14:48:02.968867 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.968830    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgfk\" (UniqueName: \"kubernetes.io/projected/70273a51-d619-48be-ab25-2caa979a630b-kube-api-access-xdgfk\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:02.969003 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.968900    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:02.969003 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.968968    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:02.969115 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.969023    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70273a51-d619-48be-ab25-2caa979a630b-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:02.969164 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:02.969120    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70273a51-d619-48be-ab25-2caa979a630b-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.070204 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.070171    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgfk\" (UniqueName: \"kubernetes.io/projected/70273a51-d619-48be-ab25-2caa979a630b-kube-api-access-xdgfk\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.070323 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.070213    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.070323 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.070235    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.070323 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.070256    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70273a51-d619-48be-ab25-2caa979a630b-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.070323 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.070306    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70273a51-d619-48be-ab25-2caa979a630b-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.070530 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:48:03.070429    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 20 14:48:03.070530 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:48:03.070489    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70273a51-d619-48be-ab25-2caa979a630b-proxy-tls podName:70273a51-d619-48be-ab25-2caa979a630b nodeName:}" failed. No retries permitted until 2026-05-20 14:48:03.570473576 +0000 UTC m=+3463.708027772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/70273a51-d619-48be-ab25-2caa979a630b-proxy-tls") pod "xgboost-v2-mlserver-predictor-7799869d6f-cldbs" (UID: "70273a51-d619-48be-ab25-2caa979a630b") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 20 14:48:03.070710 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.070691    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70273a51-d619-48be-ab25-2caa979a630b-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.070890 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.070873    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.071059 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.071040    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.082457 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.082431    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgfk\" (UniqueName: \"kubernetes.io/projected/70273a51-d619-48be-ab25-2caa979a630b-kube-api-access-xdgfk\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.216069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.216044    2557 generic.go:358] "Generic (PLEG): container finished" podID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerID="abafd989331e2bfafa5995808388fcf767163b599d45f289b746a18b21b61269" exitCode=2
May 20 14:48:03.216185 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.216125    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" event={"ID":"65fe82b6-52c9-4b82-bedb-c9504247971e","Type":"ContainerDied","Data":"abafd989331e2bfafa5995808388fcf767163b599d45f289b746a18b21b61269"}
May 20 14:48:03.576474 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.576441    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70273a51-d619-48be-ab25-2caa979a630b-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.579022 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.578985    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70273a51-d619-48be-ab25-2caa979a630b-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-cldbs\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.786217 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.786183    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:03.920196 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:03.920170    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"]
May 20 14:48:03.922296 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:48:03.922264    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70273a51_d619_48be_ab25_2caa979a630b.slice/crio-09450a6c215194b3137ddf5dc64c0c2c9eeb77213f10c05565dc64a763867618 WatchSource:0}: Error finding container 09450a6c215194b3137ddf5dc64c0c2c9eeb77213f10c05565dc64a763867618: Status 404 returned error can't find the container with id 09450a6c215194b3137ddf5dc64c0c2c9eeb77213f10c05565dc64a763867618
May 20 14:48:04.221355 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:04.221263    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" event={"ID":"70273a51-d619-48be-ab25-2caa979a630b","Type":"ContainerStarted","Data":"ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb"}
May 20 14:48:04.221355 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:04.221299    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" event={"ID":"70273a51-d619-48be-ab25-2caa979a630b","Type":"ContainerStarted","Data":"09450a6c215194b3137ddf5dc64c0c2c9eeb77213f10c05565dc64a763867618"}
May 20 14:48:05.080122 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:05.080083    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
May 20 14:48:05.086546 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:05.086518    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.57:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.57:8080: connect: connection refused"
May 20 14:48:08.235846 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.235817    2557 generic.go:358] "Generic (PLEG): container finished" podID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerID="015b310dcd669393fd715dfa5c5b5b935af2170196d3a3d134148f5ab12a92ac" exitCode=0
May 20 14:48:08.236264 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.235893    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" event={"ID":"65fe82b6-52c9-4b82-bedb-c9504247971e","Type":"ContainerDied","Data":"015b310dcd669393fd715dfa5c5b5b935af2170196d3a3d134148f5ab12a92ac"}
May 20 14:48:08.237346 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.237325    2557 generic.go:358] "Generic (PLEG): container finished" podID="70273a51-d619-48be-ab25-2caa979a630b" containerID="ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb" exitCode=0
May 20 14:48:08.237456 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.237353    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" event={"ID":"70273a51-d619-48be-ab25-2caa979a630b","Type":"ContainerDied","Data":"ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb"}
May 20 14:48:08.335335 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.335313    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:48:08.418770 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.418742    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fe82b6-52c9-4b82-bedb-c9504247971e-proxy-tls\") pod \"65fe82b6-52c9-4b82-bedb-c9504247971e\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") "
May 20 14:48:08.418939 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.418786    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-cabundle-cert\") pod \"65fe82b6-52c9-4b82-bedb-c9504247971e\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") "
May 20 14:48:08.418939 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.418838    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fe82b6-52c9-4b82-bedb-c9504247971e-kserve-provision-location\") pod \"65fe82b6-52c9-4b82-bedb-c9504247971e\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") "
May 20 14:48:08.418939 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.418907    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"65fe82b6-52c9-4b82-bedb-c9504247971e\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") "
May 20 14:48:08.419119 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.418965    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbfpb\" (UniqueName: \"kubernetes.io/projected/65fe82b6-52c9-4b82-bedb-c9504247971e-kube-api-access-tbfpb\") pod \"65fe82b6-52c9-4b82-bedb-c9504247971e\" (UID: \"65fe82b6-52c9-4b82-bedb-c9504247971e\") "
May 20 14:48:08.419172 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.419137    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fe82b6-52c9-4b82-bedb-c9504247971e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65fe82b6-52c9-4b82-bedb-c9504247971e" (UID: "65fe82b6-52c9-4b82-bedb-c9504247971e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:48:08.419210 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.419198    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65fe82b6-52c9-4b82-bedb-c9504247971e" (UID: "65fe82b6-52c9-4b82-bedb-c9504247971e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:48:08.419292 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.419271    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fe82b6-52c9-4b82-bedb-c9504247971e-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:08.419292 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.419292    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:08.419438 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.419304    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "65fe82b6-52c9-4b82-bedb-c9504247971e" (UID: "65fe82b6-52c9-4b82-bedb-c9504247971e"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:48:08.420715 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.420699    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65fe82b6-52c9-4b82-bedb-c9504247971e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65fe82b6-52c9-4b82-bedb-c9504247971e" (UID: "65fe82b6-52c9-4b82-bedb-c9504247971e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:48:08.420828 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.420815    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fe82b6-52c9-4b82-bedb-c9504247971e-kube-api-access-tbfpb" (OuterVolumeSpecName: "kube-api-access-tbfpb") pod "65fe82b6-52c9-4b82-bedb-c9504247971e" (UID: "65fe82b6-52c9-4b82-bedb-c9504247971e"). InnerVolumeSpecName "kube-api-access-tbfpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:48:08.520439 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.520412    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fe82b6-52c9-4b82-bedb-c9504247971e-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:08.520439 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.520438    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fe82b6-52c9-4b82-bedb-c9504247971e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:08.520591 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:08.520452    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfpb\" (UniqueName: \"kubernetes.io/projected/65fe82b6-52c9-4b82-bedb-c9504247971e-kube-api-access-tbfpb\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:09.242501 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.242462    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" event={"ID":"70273a51-d619-48be-ab25-2caa979a630b","Type":"ContainerStarted","Data":"117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6"}
May 20 14:48:09.242992 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.242513    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" event={"ID":"70273a51-d619-48be-ab25-2caa979a630b","Type":"ContainerStarted","Data":"129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd"}
May 20 14:48:09.242992 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.242779    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:09.244328 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.244301    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767" event={"ID":"65fe82b6-52c9-4b82-bedb-c9504247971e","Type":"ContainerDied","Data":"e85091a2630dc5fc194dbca537b2b879e2d4acae2584c1d8966ba74ba9953fd9"}
May 20 14:48:09.244328 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.244319    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"
May 20 14:48:09.244468 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.244347    2557 scope.go:117] "RemoveContainer" containerID="abafd989331e2bfafa5995808388fcf767163b599d45f289b746a18b21b61269"
May 20 14:48:09.252578 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.252562    2557 scope.go:117] "RemoveContainer" containerID="015b310dcd669393fd715dfa5c5b5b935af2170196d3a3d134148f5ab12a92ac"
May 20 14:48:09.259645 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.259628    2557 scope.go:117] "RemoveContainer" containerID="734b2014ac97b5929b4a492b101645a6660dac2b5e98cf992607446b5877f458"
May 20 14:48:09.267192 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.267150    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" podStartSLOduration=7.267133741 podStartE2EDuration="7.267133741s" podCreationTimestamp="2026-05-20 14:48:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:48:09.265842649 +0000 UTC m=+3469.403396904" watchObservedRunningTime="2026-05-20 14:48:09.267133741 +0000 UTC m=+3469.404687962"
May 20 14:48:09.281900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.281874    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"]
May 20 14:48:09.287995 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:09.287973    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-hv767"]
May 20 14:48:10.249073 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:10.249039    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:10.405009 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:10.404980    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" path="/var/lib/kubelet/pods/65fe82b6-52c9-4b82-bedb-c9504247971e/volumes"
May 20 14:48:16.257142 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:16.257109    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:46.261804 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:46.261724    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:52.983366 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:52.983320    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"]
May 20 14:48:52.983890 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:52.983736    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kserve-container" containerID="cri-o://129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd" gracePeriod=30
May 20 14:48:52.984778 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:52.984325    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kube-rbac-proxy" containerID="cri-o://117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6" gracePeriod=30
May 20 14:48:53.394632 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:53.394601    2557 generic.go:358] "Generic (PLEG): container finished" podID="70273a51-d619-48be-ab25-2caa979a630b" containerID="117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6" exitCode=2
May 20 14:48:53.394790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:53.394640    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" event={"ID":"70273a51-d619-48be-ab25-2caa979a630b","Type":"ContainerDied","Data":"117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6"}
May 20 14:48:56.252009 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:56.251969    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
May 20 14:48:56.258434 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:56.258403    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.58:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.58:8080: connect: connection refused"
May 20 14:48:58.128105 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.128084    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:58.183332 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.183260    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"70273a51-d619-48be-ab25-2caa979a630b\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") "
May 20 14:48:58.183332 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.183327    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdgfk\" (UniqueName: \"kubernetes.io/projected/70273a51-d619-48be-ab25-2caa979a630b-kube-api-access-xdgfk\") pod \"70273a51-d619-48be-ab25-2caa979a630b\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") "
May 20 14:48:58.183538 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.183381    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70273a51-d619-48be-ab25-2caa979a630b-proxy-tls\") pod \"70273a51-d619-48be-ab25-2caa979a630b\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") "
May 20 14:48:58.183538 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.183406    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70273a51-d619-48be-ab25-2caa979a630b-kserve-provision-location\") pod \"70273a51-d619-48be-ab25-2caa979a630b\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") "
May 20 14:48:58.183538 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.183441    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-cabundle-cert\") pod \"70273a51-d619-48be-ab25-2caa979a630b\" (UID: \"70273a51-d619-48be-ab25-2caa979a630b\") "
May 20 14:48:58.183706 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.183605    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "70273a51-d619-48be-ab25-2caa979a630b" (UID: "70273a51-d619-48be-ab25-2caa979a630b"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:48:58.183957 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.183928    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70273a51-d619-48be-ab25-2caa979a630b" (UID: "70273a51-d619-48be-ab25-2caa979a630b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:48:58.184082 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.183973    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70273a51-d619-48be-ab25-2caa979a630b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70273a51-d619-48be-ab25-2caa979a630b" (UID: "70273a51-d619-48be-ab25-2caa979a630b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:48:58.185503 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.185474    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70273a51-d619-48be-ab25-2caa979a630b-kube-api-access-xdgfk" (OuterVolumeSpecName: "kube-api-access-xdgfk") pod "70273a51-d619-48be-ab25-2caa979a630b" (UID: "70273a51-d619-48be-ab25-2caa979a630b"). InnerVolumeSpecName "kube-api-access-xdgfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:48:58.185581 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.185512    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70273a51-d619-48be-ab25-2caa979a630b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70273a51-d619-48be-ab25-2caa979a630b" (UID: "70273a51-d619-48be-ab25-2caa979a630b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:48:58.284523 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.284488    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:58.284523 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.284523    2557 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70273a51-d619-48be-ab25-2caa979a630b-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:58.284646 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.284534    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdgfk\" (UniqueName: \"kubernetes.io/projected/70273a51-d619-48be-ab25-2caa979a630b-kube-api-access-xdgfk\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:58.284646 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.284544    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70273a51-d619-48be-ab25-2caa979a630b-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:58.284646 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.284553    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70273a51-d619-48be-ab25-2caa979a630b-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:48:58.410761 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.410730    2557 generic.go:358] "Generic (PLEG): container finished" podID="70273a51-d619-48be-ab25-2caa979a630b" containerID="129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd" exitCode=0
May 20 14:48:58.410879 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.410807    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" event={"ID":"70273a51-d619-48be-ab25-2caa979a630b","Type":"ContainerDied","Data":"129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd"}
May 20 14:48:58.410879 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.410823    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"
May 20 14:48:58.410879 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.410838    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs" event={"ID":"70273a51-d619-48be-ab25-2caa979a630b","Type":"ContainerDied","Data":"09450a6c215194b3137ddf5dc64c0c2c9eeb77213f10c05565dc64a763867618"}
May 20 14:48:58.410879 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.410871    2557 scope.go:117] "RemoveContainer" containerID="117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6"
May 20 14:48:58.419569 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.419554    2557 scope.go:117] "RemoveContainer" containerID="129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd"
May 20 14:48:58.429218 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.429069    2557 scope.go:117] "RemoveContainer" containerID="ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb"
May 20 14:48:58.431583 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.431562    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"]
May 20 14:48:58.436640 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.436622    2557 scope.go:117] "RemoveContainer" containerID="117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6"
May 20 14:48:58.436945 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:48:58.436919    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6\": container with ID starting with 117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6 not found: ID does not exist" containerID="117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6"
May 20 14:48:58.437097 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.436949    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6"} err="failed to get container status \"117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6\": rpc error: code = NotFound desc = could not find container \"117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6\": container with ID starting with 117681e8f8a549d98c6c049e9b9bb8181aaaa4c1ac5dce39f76ef8c01751f8c6 not found: ID does not exist"
May 20 14:48:58.437097 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.436967    2557 scope.go:117] "RemoveContainer" containerID="129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd"
May 20 14:48:58.437229 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:48:58.437211    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd\": container with ID starting with 129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd not found: ID does not exist" containerID="129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd"
May 20 14:48:58.437273 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.437236    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd"} err="failed to get container status \"129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd\": rpc error: code = NotFound desc = could not find container \"129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd\": container with ID starting with 129203bad67fa754b558eec0fa6fab34723edc0e607a2c55cd0c00224e8b05bd not found: ID does not exist"
May 20 14:48:58.437273 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.437253    2557 scope.go:117] "RemoveContainer" containerID="ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb"
May 20 14:48:58.437521 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:48:58.437491    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb\": container with ID starting with ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb not found: ID does not exist" containerID="ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb"
May 20 14:48:58.437604 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.437524    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb"} err="failed to get container status \"ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb\": rpc error: code = NotFound desc = could not find container \"ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb\": container with ID starting with ccf60aa67360b1eb444ea1519122ef2377e186c312d8d41f83280b153b0cc4fb not found: ID does not exist"
May 20 14:48:58.438095 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:48:58.438075    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-cldbs"]
May 20 14:49:00.404671 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:49:00.404635    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70273a51-d619-48be-ab25-2caa979a630b" path="/var/lib/kubelet/pods/70273a51-d619-48be-ab25-2caa979a630b/volumes"
May 20 14:50:13.437508 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.437467    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"]
May 20 14:50:13.438062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.437992    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kserve-container"
May 20 14:50:13.438062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438010    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kserve-container"
May 20 14:50:13.438062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438020    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kube-rbac-proxy"
May 20 14:50:13.438062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438029    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kube-rbac-proxy"
May 20 14:50:13.438062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438050    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kube-rbac-proxy"
May 20 14:50:13.438062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438059    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kube-rbac-proxy"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438087    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="storage-initializer"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438095    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="storage-initializer"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438112    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="storage-initializer"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438119    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="storage-initializer"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438134    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kserve-container"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438150    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kserve-container"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438223    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kube-rbac-proxy"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438236    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kserve-container"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438250    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="65fe82b6-52c9-4b82-bedb-c9504247971e" containerName="kserve-container"
May 20 14:50:13.438378 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.438261    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="70273a51-d619-48be-ab25-2caa979a630b" containerName="kube-rbac-proxy"
May 20 14:50:13.444025 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.444000    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.450148 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.450117    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:50:13.450406 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.450386    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2mlds\""
May 20 14:50:13.450535 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.450407    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
May 20 14:50:13.451702 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.451675    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:50:13.452252 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.452230    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
May 20 14:50:13.452481 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.452462    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 14:50:13.454116 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.453718    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:50:13.456454 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.456429    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"]
May 20 14:50:13.543660 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.543627    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.543660 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.543662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbgr\" (UniqueName: \"kubernetes.io/projected/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kube-api-access-ztbgr\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.543917 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.543736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.543917 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.543767    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.543917 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.543796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050b37ab-aef0-4ba1-88c3-a0971a52d93f-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.644292 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.644264    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050b37ab-aef0-4ba1-88c3-a0971a52d93f-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.644415 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.644331    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.644415 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.644351    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbgr\" (UniqueName: \"kubernetes.io/projected/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kube-api-access-ztbgr\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.644415 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.644386    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.644415 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.644403    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.644810 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.644786    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.645204 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.645184    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.645251 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.645194    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.646696 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.646671    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050b37ab-aef0-4ba1-88c3-a0971a52d93f-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.653553 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.653531    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbgr\" (UniqueName: \"kubernetes.io/projected/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kube-api-access-ztbgr\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.759456 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.759370    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:13.889443 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:13.889372    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"]
May 20 14:50:13.891923 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:50:13.891891    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050b37ab_aef0_4ba1_88c3_a0971a52d93f.slice/crio-7e88f5e7b4b30c7fe1a15692b298fbeb0fc8529ce9b5c6c8c342bb9717735e6f WatchSource:0}: Error finding container 7e88f5e7b4b30c7fe1a15692b298fbeb0fc8529ce9b5c6c8c342bb9717735e6f: Status 404 returned error can't find the container with id 7e88f5e7b4b30c7fe1a15692b298fbeb0fc8529ce9b5c6c8c342bb9717735e6f
May 20 14:50:14.647420 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:14.647388    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" event={"ID":"050b37ab-aef0-4ba1-88c3-a0971a52d93f","Type":"ContainerStarted","Data":"196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22"}
May 20 14:50:14.647420 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:14.647422    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" event={"ID":"050b37ab-aef0-4ba1-88c3-a0971a52d93f","Type":"ContainerStarted","Data":"7e88f5e7b4b30c7fe1a15692b298fbeb0fc8529ce9b5c6c8c342bb9717735e6f"}
May 20 14:50:17.657398 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:17.657368    2557 generic.go:358] "Generic (PLEG): container finished" podID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerID="196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22" exitCode=0
May 20 14:50:17.657729 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:17.657447    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" event={"ID":"050b37ab-aef0-4ba1-88c3-a0971a52d93f","Type":"ContainerDied","Data":"196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22"}
May 20 14:50:18.662374 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:18.662333    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" event={"ID":"050b37ab-aef0-4ba1-88c3-a0971a52d93f","Type":"ContainerStarted","Data":"98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4"}
May 20 14:50:18.662374 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:18.662372    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" event={"ID":"050b37ab-aef0-4ba1-88c3-a0971a52d93f","Type":"ContainerStarted","Data":"6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6"}
May 20 14:50:18.662921 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:18.662611    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:18.685819 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:18.685777    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" podStartSLOduration=5.68576559 podStartE2EDuration="5.68576559s" podCreationTimestamp="2026-05-20 14:50:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:50:18.685230514 +0000 UTC m=+3598.822784734" watchObservedRunningTime="2026-05-20 14:50:18.68576559 +0000 UTC m=+3598.823319808"
May 20 14:50:19.666097 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:19.666066    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:20.619072 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:20.619042    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:50:20.623768 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:20.623744    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:50:20.627522 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:20.627498    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:50:20.632094 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:20.632067    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:50:25.677849 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:25.677818    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:50:55.689217 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:50:55.689176    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 20 14:51:05.680680 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:05.680647    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:51:13.566304 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:13.566256    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"]
May 20 14:51:13.566802 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:13.566616    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kserve-container" containerID="cri-o://6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6" gracePeriod=30
May 20 14:51:13.566802 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:13.566676    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kube-rbac-proxy" containerID="cri-o://98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4" gracePeriod=30
May 20 14:51:13.833947 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:13.833916    2557 generic.go:358] "Generic (PLEG): container finished" podID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerID="98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4" exitCode=2
May 20 14:51:13.834111 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:13.833978    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" event={"ID":"050b37ab-aef0-4ba1-88c3-a0971a52d93f","Type":"ContainerDied","Data":"98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4"}
May 20 14:51:15.672060 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:15.672016    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
May 20 14:51:19.808710 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.808686    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:51:19.854972 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.854903    2557 generic.go:358] "Generic (PLEG): container finished" podID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerID="6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6" exitCode=0
May 20 14:51:19.855114 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.854976    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"
May 20 14:51:19.855114 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.854977    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" event={"ID":"050b37ab-aef0-4ba1-88c3-a0971a52d93f","Type":"ContainerDied","Data":"6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6"}
May 20 14:51:19.855114 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.855029    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh" event={"ID":"050b37ab-aef0-4ba1-88c3-a0971a52d93f","Type":"ContainerDied","Data":"7e88f5e7b4b30c7fe1a15692b298fbeb0fc8529ce9b5c6c8c342bb9717735e6f"}
May 20 14:51:19.855114 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.855050    2557 scope.go:117] "RemoveContainer" containerID="98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4"
May 20 14:51:19.863396 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.863381    2557 scope.go:117] "RemoveContainer" containerID="6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6"
May 20 14:51:19.870351 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.870338    2557 scope.go:117] "RemoveContainer" containerID="196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22"
May 20 14:51:19.876764 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.876750    2557 scope.go:117] "RemoveContainer" containerID="98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4"
May 20 14:51:19.877088 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:51:19.877069    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4\": container with ID starting with 98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4 not found: ID does not exist" containerID="98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4"
May 20 14:51:19.877147 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.877095    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4"} err="failed to get container status \"98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4\": rpc error: code = NotFound desc = could not find container \"98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4\": container with ID starting with 98e56f3985ccbbfcdf2fd389c2945d6c3a3581fb55583bdd6a57d4cb9bf13af4 not found: ID does not exist"
May 20 14:51:19.877147 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.877109    2557 scope.go:117] "RemoveContainer" containerID="6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6"
May 20 14:51:19.877337 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:51:19.877319    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6\": container with ID starting with 6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6 not found: ID does not exist" containerID="6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6"
May 20 14:51:19.877405 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.877346    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6"} err="failed to get container status \"6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6\": rpc error: code = NotFound desc = could not find container \"6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6\": container with ID starting with 6ce38862eab1f8d1acde6d69d0fc90b19a80a838b8bf5af517995f5211387af6 not found: ID does not exist"
May 20 14:51:19.877405 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.877370    2557 scope.go:117] "RemoveContainer" containerID="196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22"
May 20 14:51:19.877566 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:51:19.877551    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22\": container with ID starting with 196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22 not found: ID does not exist" containerID="196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22"
May 20 14:51:19.877606 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.877570    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22"} err="failed to get container status \"196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22\": rpc error: code = NotFound desc = could not find container \"196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22\": container with ID starting with 196dd7a83b0c04dfb43b4a3f97a59834cb26f3d3869cdf69410b30a2ffa72e22 not found: ID does not exist"
May 20 14:51:19.933980 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.933957    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztbgr\" (UniqueName: \"kubernetes.io/projected/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kube-api-access-ztbgr\") pod \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") "
May 20 14:51:19.934065 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.933987    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-cabundle-cert\") pod \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") "
May 20 14:51:19.934065 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.934027    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") "
May 20 14:51:19.934065 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.934047    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kserve-provision-location\") pod \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") "
May 20 14:51:19.934184 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.934072    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050b37ab-aef0-4ba1-88c3-a0971a52d93f-proxy-tls\") pod \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\" (UID: \"050b37ab-aef0-4ba1-88c3-a0971a52d93f\") "
May 20 14:51:19.934391 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.934367    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "050b37ab-aef0-4ba1-88c3-a0971a52d93f" (UID: "050b37ab-aef0-4ba1-88c3-a0971a52d93f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:51:19.934501 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.934370    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "050b37ab-aef0-4ba1-88c3-a0971a52d93f" (UID: "050b37ab-aef0-4ba1-88c3-a0971a52d93f"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:51:19.934560 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.934529    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "050b37ab-aef0-4ba1-88c3-a0971a52d93f" (UID: "050b37ab-aef0-4ba1-88c3-a0971a52d93f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:51:19.935961 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.935943    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kube-api-access-ztbgr" (OuterVolumeSpecName: "kube-api-access-ztbgr") pod "050b37ab-aef0-4ba1-88c3-a0971a52d93f" (UID: "050b37ab-aef0-4ba1-88c3-a0971a52d93f"). InnerVolumeSpecName "kube-api-access-ztbgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:51:19.936039 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:19.935986    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050b37ab-aef0-4ba1-88c3-a0971a52d93f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "050b37ab-aef0-4ba1-88c3-a0971a52d93f" (UID: "050b37ab-aef0-4ba1-88c3-a0971a52d93f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:51:20.035280 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:20.035257    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbgr\" (UniqueName: \"kubernetes.io/projected/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kube-api-access-ztbgr\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:51:20.035280 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:20.035279    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:51:20.035405 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:20.035290    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/050b37ab-aef0-4ba1-88c3-a0971a52d93f-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:51:20.035405 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:20.035299    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/050b37ab-aef0-4ba1-88c3-a0971a52d93f-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:51:20.035405 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:20.035308    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/050b37ab-aef0-4ba1-88c3-a0971a52d93f-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:51:20.178954 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:20.178935    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"]
May 20 14:51:20.183033 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:20.183012    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-gctwh"]
May 20 14:51:20.405085 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:51:20.405057    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" path="/var/lib/kubelet/pods/050b37ab-aef0-4ba1-88c3-a0971a52d93f/volumes"
May 20 14:52:33.729997 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.729960    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"]
May 20 14:52:33.730469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.730316    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kserve-container"
May 20 14:52:33.730469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.730329    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kserve-container"
May 20 14:52:33.730469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.730343    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kube-rbac-proxy"
May 20 14:52:33.730469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.730348    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kube-rbac-proxy"
May 20 14:52:33.730469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.730357    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="storage-initializer"
May 20 14:52:33.730469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.730362    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="storage-initializer"
May 20 14:52:33.730469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.730417    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kube-rbac-proxy"
May 20 14:52:33.730469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.730428    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="050b37ab-aef0-4ba1-88c3-a0971a52d93f" containerName="kserve-container"
May 20 14:52:33.733383 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.733366    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.737185 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.737164    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
May 20 14:52:33.737333 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.737234    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 20 14:52:33.739895 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.739808    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:52:33.740005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.739917    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2mlds\""
May 20 14:52:33.741169 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.741149    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:52:33.741433 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.741411    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
May 20 14:52:33.743076 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.743054    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"]
May 20 14:52:33.879228 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.879209    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd61ff6-2885-488a-8cd0-733019cf32e5-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.879333 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.879238    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd61ff6-2885-488a-8cd0-733019cf32e5-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.879333 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.879261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4b2m\" (UniqueName: \"kubernetes.io/projected/3fd61ff6-2885-488a-8cd0-733019cf32e5-kube-api-access-r4b2m\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.879416 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.879382    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd61ff6-2885-488a-8cd0-733019cf32e5-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.979709 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.979682    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd61ff6-2885-488a-8cd0-733019cf32e5-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.979830 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.979722    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd61ff6-2885-488a-8cd0-733019cf32e5-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.979830 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.979741    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd61ff6-2885-488a-8cd0-733019cf32e5-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.979830 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.979761    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r4b2m\" (UniqueName: \"kubernetes.io/projected/3fd61ff6-2885-488a-8cd0-733019cf32e5-kube-api-access-r4b2m\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.980189 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.980137    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd61ff6-2885-488a-8cd0-733019cf32e5-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.980419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.980399    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd61ff6-2885-488a-8cd0-733019cf32e5-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.982219 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.982203    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd61ff6-2885-488a-8cd0-733019cf32e5-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:33.989814 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:33.989795    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4b2m\" (UniqueName: \"kubernetes.io/projected/3fd61ff6-2885-488a-8cd0-733019cf32e5-kube-api-access-r4b2m\") pod \"isvc-sklearn-s3-predictor-6c746f6458-zshfq\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:34.044636 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:34.044609    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:34.173218 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:34.173190    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"]
May 20 14:52:34.176689 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:52:34.176662    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd61ff6_2885_488a_8cd0_733019cf32e5.slice/crio-bb81f8b424a88fea9f472e97f5b84061905483613f765a25fb1e2df3ac3343ee WatchSource:0}: Error finding container bb81f8b424a88fea9f472e97f5b84061905483613f765a25fb1e2df3ac3343ee: Status 404 returned error can't find the container with id bb81f8b424a88fea9f472e97f5b84061905483613f765a25fb1e2df3ac3343ee
May 20 14:52:34.178574 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:34.178552    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:52:35.118782 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:35.118745    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" event={"ID":"3fd61ff6-2885-488a-8cd0-733019cf32e5","Type":"ContainerStarted","Data":"097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8"}
May 20 14:52:35.119199 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:35.118787    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" event={"ID":"3fd61ff6-2885-488a-8cd0-733019cf32e5","Type":"ContainerStarted","Data":"bb81f8b424a88fea9f472e97f5b84061905483613f765a25fb1e2df3ac3343ee"}
May 20 14:52:36.123737 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:36.123697    2557 generic.go:358] "Generic (PLEG): container finished" podID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerID="097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8" exitCode=0
May 20 14:52:36.124125 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:36.123790    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" event={"ID":"3fd61ff6-2885-488a-8cd0-733019cf32e5","Type":"ContainerDied","Data":"097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8"}
May 20 14:52:37.129113 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:37.129074    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" event={"ID":"3fd61ff6-2885-488a-8cd0-733019cf32e5","Type":"ContainerStarted","Data":"1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a"}
May 20 14:52:37.129113 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:37.129112    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" event={"ID":"3fd61ff6-2885-488a-8cd0-733019cf32e5","Type":"ContainerStarted","Data":"c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac"}
May 20 14:52:37.129646 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:37.129219    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:37.150422 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:37.150374    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podStartSLOduration=4.150361466 podStartE2EDuration="4.150361466s" podCreationTimestamp="2026-05-20 14:52:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:52:37.148714174 +0000 UTC m=+3737.286268416" watchObservedRunningTime="2026-05-20 14:52:37.150361466 +0000 UTC m=+3737.287915685"
May 20 14:52:38.133135 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:38.133107    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:38.134417 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:38.134381    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
May 20 14:52:39.135716 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:39.135681    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
May 20 14:52:44.140004 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:44.139973    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:52:44.140491 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:44.140464    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
May 20 14:52:54.140882 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:52:54.140821    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
May 20 14:53:04.141451 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:04.141375    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
May 20 14:53:14.140535 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:14.140498    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
May 20 14:53:24.141376 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:24.141335    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
May 20 14:53:34.140581 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:34.140545    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
May 20 14:53:44.141035 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:44.141002    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:53:53.830146 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:53.830112    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"]
May 20 14:53:53.830592 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:53.830460    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" containerID="cri-o://c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac" gracePeriod=30
May 20 14:53:53.830592 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:53.830506    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kube-rbac-proxy" containerID="cri-o://1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a" gracePeriod=30
May 20 14:53:53.955696 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:53.955666    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"]
May 20 14:53:53.959423 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:53.959404    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:53.962375 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:53.962350    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
May 20 14:53:53.962478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:53.962412    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 14:53:53.962478 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:53.962361    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
May 20 14:53:53.970056 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:53.970034    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"]
May 20 14:53:54.044395 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.044368    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.044497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.044412    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09e7696-87e3-4e65-9070-2570dafd6459-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.044497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.044430    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.044497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.044478    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ggt\" (UniqueName: \"kubernetes.io/projected/a09e7696-87e3-4e65-9070-2570dafd6459-kube-api-access-x7ggt\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.044624 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.044499    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09e7696-87e3-4e65-9070-2570dafd6459-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.136393 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.136321    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
May 20 14:53:54.141052 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.141030    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
May 20 14:53:54.145410 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.145389    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.145507 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.145428    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09e7696-87e3-4e65-9070-2570dafd6459-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.145507 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.145444    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.145605 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:53:54.145546    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
May 20 14:53:54.145605 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.145568    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ggt\" (UniqueName: \"kubernetes.io/projected/a09e7696-87e3-4e65-9070-2570dafd6459-kube-api-access-x7ggt\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.145693 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:53:54.145610    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a09e7696-87e3-4e65-9070-2570dafd6459-proxy-tls podName:a09e7696-87e3-4e65-9070-2570dafd6459 nodeName:}" failed. No retries permitted until 2026-05-20 14:53:54.64558269 +0000 UTC m=+3814.783136894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a09e7696-87e3-4e65-9070-2570dafd6459-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" (UID: "a09e7696-87e3-4e65-9070-2570dafd6459") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
May 20 14:53:54.145693 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.145652    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09e7696-87e3-4e65-9070-2570dafd6459-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.146039 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.146021    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.146089 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.146047    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09e7696-87e3-4e65-9070-2570dafd6459-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.146204 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.146186    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.154239 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.154219    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ggt\" (UniqueName: \"kubernetes.io/projected/a09e7696-87e3-4e65-9070-2570dafd6459-kube-api-access-x7ggt\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.382543 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.382516    2557 generic.go:358] "Generic (PLEG): container finished" podID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerID="1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a" exitCode=2
May 20 14:53:54.382650 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.382582    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" event={"ID":"3fd61ff6-2885-488a-8cd0-733019cf32e5","Type":"ContainerDied","Data":"1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a"}
May 20 14:53:54.649011 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.648983    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09e7696-87e3-4e65-9070-2570dafd6459-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.651389 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.651372    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09e7696-87e3-4e65-9070-2570dafd6459-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:54.869670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:54.869647    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:55.003311 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:55.003285    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"]
May 20 14:53:55.004883 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:53:55.004829    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09e7696_87e3_4e65_9070_2570dafd6459.slice/crio-ff7daecb6810cb8580ae387560b22fbb12929a8eced69d238b8ce1653bfeb70c WatchSource:0}: Error finding container ff7daecb6810cb8580ae387560b22fbb12929a8eced69d238b8ce1653bfeb70c: Status 404 returned error can't find the container with id ff7daecb6810cb8580ae387560b22fbb12929a8eced69d238b8ce1653bfeb70c
May 20 14:53:55.387461 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:55.387416    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" event={"ID":"a09e7696-87e3-4e65-9070-2570dafd6459","Type":"ContainerStarted","Data":"3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824"}
May 20 14:53:55.387668 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:55.387467    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" event={"ID":"a09e7696-87e3-4e65-9070-2570dafd6459","Type":"ContainerStarted","Data":"ff7daecb6810cb8580ae387560b22fbb12929a8eced69d238b8ce1653bfeb70c"}
May 20 14:53:56.392467 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:56.392428    2557 generic.go:358] "Generic (PLEG): container finished" podID="a09e7696-87e3-4e65-9070-2570dafd6459" containerID="3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824" exitCode=0
May 20 14:53:56.392812 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:56.392491    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" event={"ID":"a09e7696-87e3-4e65-9070-2570dafd6459","Type":"ContainerDied","Data":"3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824"}
May 20 14:53:57.397651 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.397618    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" event={"ID":"a09e7696-87e3-4e65-9070-2570dafd6459","Type":"ContainerStarted","Data":"f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95"}
May 20 14:53:57.397651 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.397654    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" event={"ID":"a09e7696-87e3-4e65-9070-2570dafd6459","Type":"ContainerStarted","Data":"20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec"}
May 20 14:53:57.398109 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.397878    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:57.419580 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.419533    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podStartSLOduration=4.419521212 podStartE2EDuration="4.419521212s" podCreationTimestamp="2026-05-20 14:53:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:53:57.418184444 +0000 UTC m=+3817.555738683" watchObservedRunningTime="2026-05-20 14:53:57.419521212 +0000 UTC m=+3817.557075430"
May 20 14:53:57.777242 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.777223    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:53:57.874353 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.874323    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd61ff6-2885-488a-8cd0-733019cf32e5-proxy-tls\") pod \"3fd61ff6-2885-488a-8cd0-733019cf32e5\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") "
May 20 14:53:57.874499 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.874389    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd61ff6-2885-488a-8cd0-733019cf32e5-kserve-provision-location\") pod \"3fd61ff6-2885-488a-8cd0-733019cf32e5\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") "
May 20 14:53:57.874499 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.874470    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4b2m\" (UniqueName: \"kubernetes.io/projected/3fd61ff6-2885-488a-8cd0-733019cf32e5-kube-api-access-r4b2m\") pod \"3fd61ff6-2885-488a-8cd0-733019cf32e5\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") "
May 20 14:53:57.874627 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.874522    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd61ff6-2885-488a-8cd0-733019cf32e5-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"3fd61ff6-2885-488a-8cd0-733019cf32e5\" (UID: \"3fd61ff6-2885-488a-8cd0-733019cf32e5\") "
May 20 14:53:57.874681 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.874631    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd61ff6-2885-488a-8cd0-733019cf32e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fd61ff6-2885-488a-8cd0-733019cf32e5" (UID: "3fd61ff6-2885-488a-8cd0-733019cf32e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:53:57.874774 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.874758    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fd61ff6-2885-488a-8cd0-733019cf32e5-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:53:57.875042 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.875005    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd61ff6-2885-488a-8cd0-733019cf32e5-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "3fd61ff6-2885-488a-8cd0-733019cf32e5" (UID: "3fd61ff6-2885-488a-8cd0-733019cf32e5"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:53:57.876459 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.876440    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd61ff6-2885-488a-8cd0-733019cf32e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fd61ff6-2885-488a-8cd0-733019cf32e5" (UID: "3fd61ff6-2885-488a-8cd0-733019cf32e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:53:57.876559 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.876527    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd61ff6-2885-488a-8cd0-733019cf32e5-kube-api-access-r4b2m" (OuterVolumeSpecName: "kube-api-access-r4b2m") pod "3fd61ff6-2885-488a-8cd0-733019cf32e5" (UID: "3fd61ff6-2885-488a-8cd0-733019cf32e5"). InnerVolumeSpecName "kube-api-access-r4b2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:53:57.975490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.975423    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4b2m\" (UniqueName: \"kubernetes.io/projected/3fd61ff6-2885-488a-8cd0-733019cf32e5-kube-api-access-r4b2m\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:53:57.975490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.975452    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fd61ff6-2885-488a-8cd0-733019cf32e5-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:53:57.975490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:57.975467    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fd61ff6-2885-488a-8cd0-733019cf32e5-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:53:58.405719 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.405688    2557 generic.go:358] "Generic (PLEG): container finished" podID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerID="c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac" exitCode=0
May 20 14:53:58.406196 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.405839    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"
May 20 14:53:58.407672 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.407642    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
May 20 14:53:58.408031 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.408015    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:53:58.408102 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.408037    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" event={"ID":"3fd61ff6-2885-488a-8cd0-733019cf32e5","Type":"ContainerDied","Data":"c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac"}
May 20 14:53:58.408102 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.408056    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq" event={"ID":"3fd61ff6-2885-488a-8cd0-733019cf32e5","Type":"ContainerDied","Data":"bb81f8b424a88fea9f472e97f5b84061905483613f765a25fb1e2df3ac3343ee"}
May 20 14:53:58.408102 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.408073    2557 scope.go:117] "RemoveContainer" containerID="1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a"
May 20 14:53:58.416790 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.416765    2557 scope.go:117] "RemoveContainer" containerID="c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac"
May 20 14:53:58.424825 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.424806    2557 scope.go:117] "RemoveContainer" containerID="097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8"
May 20 14:53:58.432408 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.432390    2557 scope.go:117] "RemoveContainer" containerID="1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a"
May 20 14:53:58.432471 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.432434    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"]
May 20 14:53:58.432682 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:53:58.432662    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a\": container with ID starting with 1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a not found: ID does not exist" containerID="1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a"
May 20 14:53:58.432744 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.432689    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a"} err="failed to get container status \"1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a\": rpc error: code = NotFound desc = could not find container \"1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a\": container with ID starting with 1e16669e567a595b46ac94c9200044770c8bd1ed60d42da03208af1f06e4bc8a not found: ID does not exist"
May 20 14:53:58.432744 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.432706    2557 scope.go:117] "RemoveContainer" containerID="c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac"
May 20 14:53:58.432985 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:53:58.432964    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac\": container with ID starting with c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac not found: ID does not exist" containerID="c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac"
May 20 14:53:58.433044 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.432990    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac"} err="failed to get container status \"c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac\": rpc error: code = NotFound desc = could not find container \"c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac\": container with ID starting with c3547e1899cc3d15122801fa4b957ee84f9db02f7a7c74570e9ca194a2d98fac not found: ID does not exist"
May 20 14:53:58.433044 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.433008    2557 scope.go:117] "RemoveContainer" containerID="097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8"
May 20 14:53:58.433238 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:53:58.433222    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8\": container with ID starting with 097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8 not found: ID does not exist" containerID="097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8"
May 20 14:53:58.433282 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.433242    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8"} err="failed to get container status \"097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8\": rpc error: code = NotFound desc = could not find container \"097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8\": container with ID starting with 097fd39e8d0d5bc212f18e2bae78de6b581d31760dcb09e6d48cf06edd615ed8 not found: ID does not exist"
May 20 14:53:58.439366 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:58.439348    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6c746f6458-zshfq"]
May 20 14:53:59.410344 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:53:59.410302    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
May 20 14:54:00.406207 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:54:00.406172    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" path="/var/lib/kubelet/pods/3fd61ff6-2885-488a-8cd0-733019cf32e5/volumes"
May 20 14:54:04.415126 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:54:04.415091    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:54:04.415720 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:54:04.415689    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
May 20 14:54:14.415883 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:54:14.415823    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
May 20 14:54:24.416538 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:54:24.416505    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
May 20 14:54:34.416141 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:54:34.416059    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
May 20 14:54:44.416233 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:54:44.416195    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
May 20 14:54:54.416608 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:54:54.416566    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
May 20 14:55:04.416043 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:04.416008    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:55:13.980525 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:13.980485    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"]
May 20 14:55:13.981053 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:13.980841    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" containerID="cri-o://20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec" gracePeriod=30
May 20 14:55:13.981053 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:13.980896    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kube-rbac-proxy" containerID="cri-o://f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95" gracePeriod=30
May 20 14:55:14.410621 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:14.410583    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
May 20 14:55:14.415965 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:14.415940    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
May 20 14:55:14.656309 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:14.656274    2557 generic.go:358] "Generic (PLEG): container finished" podID="a09e7696-87e3-4e65-9070-2570dafd6459" containerID="f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95" exitCode=2
May 20 14:55:14.656441 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:14.656342    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" event={"ID":"a09e7696-87e3-4e65-9070-2570dafd6459","Type":"ContainerDied","Data":"f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95"}
May 20 14:55:15.054609 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.054571    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"]
May 20 14:55:15.055170 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.055150    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="storage-initializer"
May 20 14:55:15.055258 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.055172    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="storage-initializer"
May 20 14:55:15.055258 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.055185    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container"
May 20 14:55:15.055258 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.055193    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container"
May 20 14:55:15.055258 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.055215    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kube-rbac-proxy"
May 20 14:55:15.055258 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.055223    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kube-rbac-proxy"
May 20 14:55:15.055524 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.055298    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kserve-container"
May 20 14:55:15.055524 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.055316    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="3fd61ff6-2885-488a-8cd0-733019cf32e5" containerName="kube-rbac-proxy"
May 20 14:55:15.058444 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.058423    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.061409 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.061392    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
May 20 14:55:15.061689 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.061675    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
May 20 14:55:15.070532 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.070510    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"]
May 20 14:55:15.141562 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.141531    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvch\" (UniqueName: \"kubernetes.io/projected/93a675b9-1c93-4659-b6be-3f7d37525d7e-kube-api-access-dsvch\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.141695 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.141584    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a675b9-1c93-4659-b6be-3f7d37525d7e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.141695 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.141631    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a675b9-1c93-4659-b6be-3f7d37525d7e-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.141776 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.141701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a675b9-1c93-4659-b6be-3f7d37525d7e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.242649 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.242614    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a675b9-1c93-4659-b6be-3f7d37525d7e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.242798 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.242657    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a675b9-1c93-4659-b6be-3f7d37525d7e-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.242798 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.242725    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a675b9-1c93-4659-b6be-3f7d37525d7e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.242798 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.242765    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvch\" (UniqueName: \"kubernetes.io/projected/93a675b9-1c93-4659-b6be-3f7d37525d7e-kube-api-access-dsvch\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.242798 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:55:15.242778    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
May 20 14:55:15.243034 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:55:15.242880    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a675b9-1c93-4659-b6be-3f7d37525d7e-proxy-tls podName:93a675b9-1c93-4659-b6be-3f7d37525d7e nodeName:}" failed. No retries permitted until 2026-05-20 14:55:15.742834842 +0000 UTC m=+3895.880389054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/93a675b9-1c93-4659-b6be-3f7d37525d7e-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr" (UID: "93a675b9-1c93-4659-b6be-3f7d37525d7e") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
May 20 14:55:15.243207 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.243181    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a675b9-1c93-4659-b6be-3f7d37525d7e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.243403 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.243387    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a675b9-1c93-4659-b6be-3f7d37525d7e-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.251620 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.251595    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvch\" (UniqueName: \"kubernetes.io/projected/93a675b9-1c93-4659-b6be-3f7d37525d7e-kube-api-access-dsvch\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.746037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.745994    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a675b9-1c93-4659-b6be-3f7d37525d7e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.748347 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.748317    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a675b9-1c93-4659-b6be-3f7d37525d7e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:15.968906 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:15.968873    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:16.095076 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:16.095046    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"]
May 20 14:55:16.097879 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:55:16.097829    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a675b9_1c93_4659_b6be_3f7d37525d7e.slice/crio-80ea3c98743d8c3326a27d87f0cf66c2ee7b5f581a3ebf1a4901f02640445151 WatchSource:0}: Error finding container 80ea3c98743d8c3326a27d87f0cf66c2ee7b5f581a3ebf1a4901f02640445151: Status 404 returned error can't find the container with id 80ea3c98743d8c3326a27d87f0cf66c2ee7b5f581a3ebf1a4901f02640445151
May 20 14:55:16.664082 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:16.664041    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr" event={"ID":"93a675b9-1c93-4659-b6be-3f7d37525d7e","Type":"ContainerStarted","Data":"aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367"}
May 20 14:55:16.664241 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:16.664092    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr" event={"ID":"93a675b9-1c93-4659-b6be-3f7d37525d7e","Type":"ContainerStarted","Data":"80ea3c98743d8c3326a27d87f0cf66c2ee7b5f581a3ebf1a4901f02640445151"}
May 20 14:55:18.023249 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.023226    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:55:18.169196 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.169171    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-cabundle-cert\") pod \"a09e7696-87e3-4e65-9070-2570dafd6459\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") "
May 20 14:55:18.169318 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.169212    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09e7696-87e3-4e65-9070-2570dafd6459-proxy-tls\") pod \"a09e7696-87e3-4e65-9070-2570dafd6459\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") "
May 20 14:55:18.169318 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.169249    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09e7696-87e3-4e65-9070-2570dafd6459-kserve-provision-location\") pod \"a09e7696-87e3-4e65-9070-2570dafd6459\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") "
May 20 14:55:18.169318 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.169273    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"a09e7696-87e3-4e65-9070-2570dafd6459\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") "
May 20 14:55:18.169318 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.169290    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7ggt\" (UniqueName: \"kubernetes.io/projected/a09e7696-87e3-4e65-9070-2570dafd6459-kube-api-access-x7ggt\") pod \"a09e7696-87e3-4e65-9070-2570dafd6459\" (UID: \"a09e7696-87e3-4e65-9070-2570dafd6459\") "
May 20 14:55:18.169617 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.169591    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a09e7696-87e3-4e65-9070-2570dafd6459" (UID: "a09e7696-87e3-4e65-9070-2570dafd6459"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:55:18.169677 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.169629    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "a09e7696-87e3-4e65-9070-2570dafd6459" (UID: "a09e7696-87e3-4e65-9070-2570dafd6459"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:55:18.169677 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.169640    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09e7696-87e3-4e65-9070-2570dafd6459-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a09e7696-87e3-4e65-9070-2570dafd6459" (UID: "a09e7696-87e3-4e65-9070-2570dafd6459"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:55:18.171343 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.171317    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09e7696-87e3-4e65-9070-2570dafd6459-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a09e7696-87e3-4e65-9070-2570dafd6459" (UID: "a09e7696-87e3-4e65-9070-2570dafd6459"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:55:18.171488 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.171463    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09e7696-87e3-4e65-9070-2570dafd6459-kube-api-access-x7ggt" (OuterVolumeSpecName: "kube-api-access-x7ggt") pod "a09e7696-87e3-4e65-9070-2570dafd6459" (UID: "a09e7696-87e3-4e65-9070-2570dafd6459"). InnerVolumeSpecName "kube-api-access-x7ggt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:55:18.270029 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.270001    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:55:18.270029 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.270027    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ggt\" (UniqueName: \"kubernetes.io/projected/a09e7696-87e3-4e65-9070-2570dafd6459-kube-api-access-x7ggt\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:55:18.270198 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.270043    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a09e7696-87e3-4e65-9070-2570dafd6459-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:55:18.270198 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.270058    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a09e7696-87e3-4e65-9070-2570dafd6459-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:55:18.270198 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.270072    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a09e7696-87e3-4e65-9070-2570dafd6459-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:55:18.673581 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.673549    2557 generic.go:358] "Generic (PLEG): container finished" podID="a09e7696-87e3-4e65-9070-2570dafd6459" containerID="20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec" exitCode=0
May 20 14:55:18.673686 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.673595    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" event={"ID":"a09e7696-87e3-4e65-9070-2570dafd6459","Type":"ContainerDied","Data":"20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec"}
May 20 14:55:18.673686 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.673629    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs" event={"ID":"a09e7696-87e3-4e65-9070-2570dafd6459","Type":"ContainerDied","Data":"ff7daecb6810cb8580ae387560b22fbb12929a8eced69d238b8ce1653bfeb70c"}
May 20 14:55:18.673686 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.673645    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"
May 20 14:55:18.673686 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.673651    2557 scope.go:117] "RemoveContainer" containerID="f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95"
May 20 14:55:18.681455 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.681432    2557 scope.go:117] "RemoveContainer" containerID="20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec"
May 20 14:55:18.688361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.688347    2557 scope.go:117] "RemoveContainer" containerID="3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824"
May 20 14:55:18.692565 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.692540    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"]
May 20 14:55:18.696362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.696346    2557 scope.go:117] "RemoveContainer" containerID="f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95"
May 20 14:55:18.696652 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:55:18.696631    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95\": container with ID starting with f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95 not found: ID does not exist" containerID="f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95"
May 20 14:55:18.696753 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.696657    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95"} err="failed to get container status \"f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95\": rpc error: code = NotFound desc = could not find container \"f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95\": container with ID starting with f8cbb9e03b21a9c68a86c32fe3bbdc480af5dfac1c6692166759ceb4d83aaf95 not found: ID does not exist"
May 20 14:55:18.696753 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.696677    2557 scope.go:117] "RemoveContainer" containerID="20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec"
May 20 14:55:18.697081 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:55:18.697051    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec\": container with ID starting with 20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec not found: ID does not exist" containerID="20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec"
May 20 14:55:18.697163 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.697084    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec"} err="failed to get container status \"20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec\": rpc error: code = NotFound desc = could not find container \"20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec\": container with ID starting with 20af3a4d6ef7cba351d39a4adeec64af1068d3b3221c7590e201cdb24509a0ec not found: ID does not exist"
May 20 14:55:18.697163 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.697103    2557 scope.go:117] "RemoveContainer" containerID="3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824"
May 20 14:55:18.697326 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.697306    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-59d77d6cd-q5jjs"]
May 20 14:55:18.697326 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:55:18.697315    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824\": container with ID starting with 3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824 not found: ID does not exist" containerID="3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824"
May 20 14:55:18.697415 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:18.697344    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824"} err="failed to get container status \"3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824\": rpc error: code = NotFound desc = could not find container \"3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824\": container with ID starting with 3c20d636300284bd968eb6f8fa0c7c744e7da672b0d18473a08adf4b9578f824 not found: ID does not exist"
May 20 14:55:20.406486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:20.406447    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" path="/var/lib/kubelet/pods/a09e7696-87e3-4e65-9070-2570dafd6459/volumes"
May 20 14:55:20.658027 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:20.657949    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:55:20.662086 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:20.662065    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:55:20.662540 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:20.662520    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:55:20.666621 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:20.666601    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:55:21.686302 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:21.686278    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr_93a675b9-1c93-4659-b6be-3f7d37525d7e/storage-initializer/0.log"
May 20 14:55:21.686652 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:21.686313    2557 generic.go:358] "Generic (PLEG): container finished" podID="93a675b9-1c93-4659-b6be-3f7d37525d7e" containerID="aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367" exitCode=1
May 20 14:55:21.686652 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:21.686397    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr" event={"ID":"93a675b9-1c93-4659-b6be-3f7d37525d7e","Type":"ContainerDied","Data":"aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367"}
May 20 14:55:22.692333 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:22.692308    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr_93a675b9-1c93-4659-b6be-3f7d37525d7e/storage-initializer/0.log"
May 20 14:55:22.692693 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:22.692427    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr" event={"ID":"93a675b9-1c93-4659-b6be-3f7d37525d7e","Type":"ContainerStarted","Data":"a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add"}
May 20 14:55:25.140703 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.140672    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"]
May 20 14:55:25.141113 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.140929    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr" podUID="93a675b9-1c93-4659-b6be-3f7d37525d7e" containerName="storage-initializer" containerID="cri-o://a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add" gracePeriod=30
May 20 14:55:25.580339 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.580319    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr_93a675b9-1c93-4659-b6be-3f7d37525d7e/storage-initializer/1.log"
May 20 14:55:25.580708 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.580694    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr_93a675b9-1c93-4659-b6be-3f7d37525d7e/storage-initializer/0.log"
May 20 14:55:25.580771 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.580760    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:25.703747 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.703694    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr_93a675b9-1c93-4659-b6be-3f7d37525d7e/storage-initializer/1.log"
May 20 14:55:25.704037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.704024    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr_93a675b9-1c93-4659-b6be-3f7d37525d7e/storage-initializer/0.log"
May 20 14:55:25.704099 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.704055    2557 generic.go:358] "Generic (PLEG): container finished" podID="93a675b9-1c93-4659-b6be-3f7d37525d7e" containerID="a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add" exitCode=1
May 20 14:55:25.704157 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.704140    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"
May 20 14:55:25.704195 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.704155    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr" event={"ID":"93a675b9-1c93-4659-b6be-3f7d37525d7e","Type":"ContainerDied","Data":"a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add"}
May 20 14:55:25.704235 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.704196    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr" event={"ID":"93a675b9-1c93-4659-b6be-3f7d37525d7e","Type":"ContainerDied","Data":"80ea3c98743d8c3326a27d87f0cf66c2ee7b5f581a3ebf1a4901f02640445151"}
May 20 14:55:25.704235 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.704217    2557 scope.go:117] "RemoveContainer" containerID="a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add"
May 20 14:55:25.711662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.711646    2557 scope.go:117] "RemoveContainer" containerID="aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367"
May 20 14:55:25.718632 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.718615    2557 scope.go:117] "RemoveContainer" containerID="a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add"
May 20 14:55:25.718900 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:55:25.718848    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add\": container with ID starting with a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add not found: ID does not exist" containerID="a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add"
May 20 14:55:25.718900 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.718889    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add"} err="failed to get container status \"a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add\": rpc error: code = NotFound desc = could not find container \"a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add\": container with ID starting with a1a365f47222f2cb32ec0d152ff85d6a1361358bf90e52f188090d5fb15d6add not found: ID does not exist"
May 20 14:55:25.719018 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.718907    2557 scope.go:117] "RemoveContainer" containerID="aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367"
May 20 14:55:25.719098 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:55:25.719084    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367\": container with ID starting with aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367 not found: ID does not exist" containerID="aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367"
May 20 14:55:25.719136 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.719101    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367"} err="failed to get container status \"aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367\": rpc error: code = NotFound desc = could not find container \"aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367\": container with ID starting with aacf9734ff15e19e13245b222133cb447ace1291031ac95805380390c4526367 not found: ID does not exist"
May 20 14:55:25.728382 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.728366    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a675b9-1c93-4659-b6be-3f7d37525d7e-proxy-tls\") pod \"93a675b9-1c93-4659-b6be-3f7d37525d7e\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") "
May 20 14:55:25.728441 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.728400    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a675b9-1c93-4659-b6be-3f7d37525d7e-kserve-provision-location\") pod \"93a675b9-1c93-4659-b6be-3f7d37525d7e\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") "
May 20 14:55:25.728494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.728448    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a675b9-1c93-4659-b6be-3f7d37525d7e-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"93a675b9-1c93-4659-b6be-3f7d37525d7e\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") "
May 20 14:55:25.728494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.728478    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsvch\" (UniqueName: \"kubernetes.io/projected/93a675b9-1c93-4659-b6be-3f7d37525d7e-kube-api-access-dsvch\") pod \"93a675b9-1c93-4659-b6be-3f7d37525d7e\" (UID: \"93a675b9-1c93-4659-b6be-3f7d37525d7e\") "
May 20 14:55:25.728746 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.728719    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a675b9-1c93-4659-b6be-3f7d37525d7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93a675b9-1c93-4659-b6be-3f7d37525d7e" (UID: "93a675b9-1c93-4659-b6be-3f7d37525d7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:55:25.728830 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.728765    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a675b9-1c93-4659-b6be-3f7d37525d7e-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "93a675b9-1c93-4659-b6be-3f7d37525d7e" (UID: "93a675b9-1c93-4659-b6be-3f7d37525d7e"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:55:25.730236 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.730218    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a675b9-1c93-4659-b6be-3f7d37525d7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93a675b9-1c93-4659-b6be-3f7d37525d7e" (UID: "93a675b9-1c93-4659-b6be-3f7d37525d7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:55:25.730400 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.730381    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a675b9-1c93-4659-b6be-3f7d37525d7e-kube-api-access-dsvch" (OuterVolumeSpecName: "kube-api-access-dsvch") pod "93a675b9-1c93-4659-b6be-3f7d37525d7e" (UID: "93a675b9-1c93-4659-b6be-3f7d37525d7e"). InnerVolumeSpecName "kube-api-access-dsvch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:55:25.829969 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.829945    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsvch\" (UniqueName: \"kubernetes.io/projected/93a675b9-1c93-4659-b6be-3f7d37525d7e-kube-api-access-dsvch\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:55:25.829969 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.829967    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a675b9-1c93-4659-b6be-3f7d37525d7e-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:55:25.830094 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.829983    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a675b9-1c93-4659-b6be-3f7d37525d7e-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:55:25.830094 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:25.829999    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a675b9-1c93-4659-b6be-3f7d37525d7e-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:55:26.045214 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.045153    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"]
May 20 14:55:26.047646 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.047618    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5545d8ff9b-ts4vr"]
May 20 14:55:26.152137 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152106    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"]
May 20 14:55:26.152467 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152435    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container"
May 20 14:55:26.152467 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152445    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container"
May 20 14:55:26.152467 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152457    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a675b9-1c93-4659-b6be-3f7d37525d7e" containerName="storage-initializer"
May 20 14:55:26.152467 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152463    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a675b9-1c93-4659-b6be-3f7d37525d7e" containerName="storage-initializer"
May 20 14:55:26.152585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152484    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a675b9-1c93-4659-b6be-3f7d37525d7e" containerName="storage-initializer"
May 20 14:55:26.152585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152490    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a675b9-1c93-4659-b6be-3f7d37525d7e" containerName="storage-initializer"
May 20 14:55:26.152585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152497    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kube-rbac-proxy"
May 20 14:55:26.152585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152502    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kube-rbac-proxy"
May 20 14:55:26.152585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152513    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="storage-initializer"
May 20 14:55:26.152585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152518    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="storage-initializer"
May 20 14:55:26.152585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152570    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="93a675b9-1c93-4659-b6be-3f7d37525d7e" containerName="storage-initializer"
May 20 14:55:26.152585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152580    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kube-rbac-proxy"
May 20 14:55:26.152585 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152588    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="a09e7696-87e3-4e65-9070-2570dafd6459" containerName="kserve-container"
May 20 14:55:26.152887 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.152703    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="93a675b9-1c93-4659-b6be-3f7d37525d7e" containerName="storage-initializer"
May 20 14:55:26.157241 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.157223    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.160418 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.160399    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
May 20 14:55:26.160978 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.160952    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:55:26.161088 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.160958    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2mlds\""
May 20 14:55:26.161088 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.160953    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
May 20 14:55:26.161302 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.161284    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 20 14:55:26.161374 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.161310    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:55:26.161374 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.161284    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 14:55:26.168581 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.168559    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"]
May 20 14:55:26.334763 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.334740    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4dd34e-df90-45dd-bba9-65c3a5507d66-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.334887 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.334770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.334887 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.334803    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.334988 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.334921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvmrs\" (UniqueName: \"kubernetes.io/projected/be4dd34e-df90-45dd-bba9-65c3a5507d66-kube-api-access-kvmrs\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.335023 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.334987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4dd34e-df90-45dd-bba9-65c3a5507d66-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.406341 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.406312    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a675b9-1c93-4659-b6be-3f7d37525d7e" path="/var/lib/kubelet/pods/93a675b9-1c93-4659-b6be-3f7d37525d7e/volumes"
May 20 14:55:26.435463 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.435439    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4dd34e-df90-45dd-bba9-65c3a5507d66-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.435561 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.435469    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.435561 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.435492    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.435561 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.435542    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvmrs\" (UniqueName: \"kubernetes.io/projected/be4dd34e-df90-45dd-bba9-65c3a5507d66-kube-api-access-kvmrs\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.435702 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.435578    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4dd34e-df90-45dd-bba9-65c3a5507d66-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.436008 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.435980    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4dd34e-df90-45dd-bba9-65c3a5507d66-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.436274 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.436250    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.436327 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.436253    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.438009 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.437987    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4dd34e-df90-45dd-bba9-65c3a5507d66-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.445302 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.445283    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvmrs\" (UniqueName: \"kubernetes.io/projected/be4dd34e-df90-45dd-bba9-65c3a5507d66-kube-api-access-kvmrs\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.467869 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.467833    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:26.596415 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.596391    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"]
May 20 14:55:26.599048 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:55:26.599019    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4dd34e_df90_45dd_bba9_65c3a5507d66.slice/crio-4471cce93c6bc9a82d187a92f2ce681cf1bac8761b497c1fc855d7cfd9ec58c8 WatchSource:0}: Error finding container 4471cce93c6bc9a82d187a92f2ce681cf1bac8761b497c1fc855d7cfd9ec58c8: Status 404 returned error can't find the container with id 4471cce93c6bc9a82d187a92f2ce681cf1bac8761b497c1fc855d7cfd9ec58c8
May 20 14:55:26.709501 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.709473    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" event={"ID":"be4dd34e-df90-45dd-bba9-65c3a5507d66","Type":"ContainerStarted","Data":"a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1"}
May 20 14:55:26.709501 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:26.709504    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" event={"ID":"be4dd34e-df90-45dd-bba9-65c3a5507d66","Type":"ContainerStarted","Data":"4471cce93c6bc9a82d187a92f2ce681cf1bac8761b497c1fc855d7cfd9ec58c8"}
May 20 14:55:27.714361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:27.714327    2557 generic.go:358] "Generic (PLEG): container finished" podID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerID="a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1" exitCode=0
May 20 14:55:27.714749 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:27.714366    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" event={"ID":"be4dd34e-df90-45dd-bba9-65c3a5507d66","Type":"ContainerDied","Data":"a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1"}
May 20 14:55:28.719682 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:28.719643    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" event={"ID":"be4dd34e-df90-45dd-bba9-65c3a5507d66","Type":"ContainerStarted","Data":"72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba"}
May 20 14:55:28.719682 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:28.719686    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" event={"ID":"be4dd34e-df90-45dd-bba9-65c3a5507d66","Type":"ContainerStarted","Data":"6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769"}
May 20 14:55:28.720164 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:28.719717    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:28.742894 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:28.742830    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podStartSLOduration=2.742815135 podStartE2EDuration="2.742815135s" podCreationTimestamp="2026-05-20 14:55:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:55:28.740451913 +0000 UTC m=+3908.878006133" watchObservedRunningTime="2026-05-20 14:55:28.742815135 +0000 UTC m=+3908.880369386"
May 20 14:55:29.722786 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:29.722754    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:29.723781 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:29.723748    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 20 14:55:30.726452 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:30.726410    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 20 14:55:35.731430 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:35.731395    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:55:35.732018 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:35.731989    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 20 14:55:45.731977 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:45.731930    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 20 14:55:55.732776 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:55:55.732740    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 20 14:56:05.732314 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:05.732216    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 20 14:56:15.731989 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:15.731950    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 20 14:56:25.732434 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:25.732392    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 20 14:56:35.733031 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:35.732995    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:56:36.175539 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:36.175499    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"]
May 20 14:56:36.175991 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:36.175953    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container" containerID="cri-o://6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769" gracePeriod=30
May 20 14:56:36.176192 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:36.176149    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kube-rbac-proxy" containerID="cri-o://72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba" gracePeriod=30
May 20 14:56:36.937465 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:36.937427    2557 generic.go:358] "Generic (PLEG): container finished" podID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerID="72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba" exitCode=2
May 20 14:56:36.937825 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:36.937491    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" event={"ID":"be4dd34e-df90-45dd-bba9-65c3a5507d66","Type":"ContainerDied","Data":"72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba"}
May 20 14:56:37.262543 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.262468    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"]
May 20 14:56:37.265935 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.265918    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.268447 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.268420    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
May 20 14:56:37.268565 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.268473    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
May 20 14:56:37.281055 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.281034    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"]
May 20 14:56:37.341048 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.341019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c1e5a2-53c8-4854-bb20-42cb6a219182-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.341186 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.341053    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c1e5a2-53c8-4854-bb20-42cb6a219182-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.341186 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.341150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.341288 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.341192    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjlv\" (UniqueName: \"kubernetes.io/projected/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kube-api-access-2qjlv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.442107 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.442077    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.442228 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.442120    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjlv\" (UniqueName: \"kubernetes.io/projected/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kube-api-access-2qjlv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.442228 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.442166    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c1e5a2-53c8-4854-bb20-42cb6a219182-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.442228 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.442200    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c1e5a2-53c8-4854-bb20-42cb6a219182-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.442409 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:56:37.442307    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
May 20 14:56:37.442409 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:56:37.442365    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1e5a2-53c8-4854-bb20-42cb6a219182-proxy-tls podName:c1c1e5a2-53c8-4854-bb20-42cb6a219182 nodeName:}" failed. No retries permitted until 2026-05-20 14:56:37.942345626 +0000 UTC m=+3978.079899840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c1c1e5a2-53c8-4854-bb20-42cb6a219182-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv" (UID: "c1c1e5a2-53c8-4854-bb20-42cb6a219182") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
May 20 14:56:37.442515 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.442485    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.442807 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.442787    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c1e5a2-53c8-4854-bb20-42cb6a219182-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.452441 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.452413    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjlv\" (UniqueName: \"kubernetes.io/projected/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kube-api-access-2qjlv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.946175 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.946142    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c1e5a2-53c8-4854-bb20-42cb6a219182-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:37.948623 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:37.948592    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c1e5a2-53c8-4854-bb20-42cb6a219182-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:38.178052 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:38.178022    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:38.307632 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:38.307608    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"]
May 20 14:56:38.309432 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:56:38.309404    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c1e5a2_53c8_4854_bb20_42cb6a219182.slice/crio-638ce94dd8b5519e8c3c486c02164a8afe2a64c392a636ba26c8dfd80626662c WatchSource:0}: Error finding container 638ce94dd8b5519e8c3c486c02164a8afe2a64c392a636ba26c8dfd80626662c: Status 404 returned error can't find the container with id 638ce94dd8b5519e8c3c486c02164a8afe2a64c392a636ba26c8dfd80626662c
May 20 14:56:38.945192 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:38.945154    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv" event={"ID":"c1c1e5a2-53c8-4854-bb20-42cb6a219182","Type":"ContainerStarted","Data":"2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306"}
May 20 14:56:38.945192 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:38.945196    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv" event={"ID":"c1c1e5a2-53c8-4854-bb20-42cb6a219182","Type":"ContainerStarted","Data":"638ce94dd8b5519e8c3c486c02164a8afe2a64c392a636ba26c8dfd80626662c"}
May 20 14:56:39.921760 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.921730    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:56:39.949434 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.949370    2557 generic.go:358] "Generic (PLEG): container finished" podID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerID="6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769" exitCode=0
May 20 14:56:39.949526 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.949457    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"
May 20 14:56:39.949526 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.949457    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" event={"ID":"be4dd34e-df90-45dd-bba9-65c3a5507d66","Type":"ContainerDied","Data":"6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769"}
May 20 14:56:39.949526 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.949494    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v" event={"ID":"be4dd34e-df90-45dd-bba9-65c3a5507d66","Type":"ContainerDied","Data":"4471cce93c6bc9a82d187a92f2ce681cf1bac8761b497c1fc855d7cfd9ec58c8"}
May 20 14:56:39.949526 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.949511    2557 scope.go:117] "RemoveContainer" containerID="72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba"
May 20 14:56:39.958472 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.958450    2557 scope.go:117] "RemoveContainer" containerID="6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769"
May 20 14:56:39.964543 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.964522    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-cabundle-cert\") pod \"be4dd34e-df90-45dd-bba9-65c3a5507d66\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") "
May 20 14:56:39.964626 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.964578    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvmrs\" (UniqueName: \"kubernetes.io/projected/be4dd34e-df90-45dd-bba9-65c3a5507d66-kube-api-access-kvmrs\") pod \"be4dd34e-df90-45dd-bba9-65c3a5507d66\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") "
May 20 14:56:39.964691 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.964622    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4dd34e-df90-45dd-bba9-65c3a5507d66-proxy-tls\") pod \"be4dd34e-df90-45dd-bba9-65c3a5507d66\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") "
May 20 14:56:39.964691 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.964657    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"be4dd34e-df90-45dd-bba9-65c3a5507d66\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") "
May 20 14:56:39.965281 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.964848    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4dd34e-df90-45dd-bba9-65c3a5507d66-kserve-provision-location\") pod \"be4dd34e-df90-45dd-bba9-65c3a5507d66\" (UID: \"be4dd34e-df90-45dd-bba9-65c3a5507d66\") "
May 20 14:56:39.965281 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.964954    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be4dd34e-df90-45dd-bba9-65c3a5507d66" (UID: "be4dd34e-df90-45dd-bba9-65c3a5507d66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:56:39.965281 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.965040    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "be4dd34e-df90-45dd-bba9-65c3a5507d66" (UID: "be4dd34e-df90-45dd-bba9-65c3a5507d66"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:56:39.965281 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.965124    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4dd34e-df90-45dd-bba9-65c3a5507d66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be4dd34e-df90-45dd-bba9-65c3a5507d66" (UID: "be4dd34e-df90-45dd-bba9-65c3a5507d66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:56:39.965281 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.965219    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:56:39.965281 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.965241    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be4dd34e-df90-45dd-bba9-65c3a5507d66-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:56:39.965281 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.965259    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4dd34e-df90-45dd-bba9-65c3a5507d66-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:56:39.966395 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.966371    2557 scope.go:117] "RemoveContainer" containerID="a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1"
May 20 14:56:39.966793 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.966760    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4dd34e-df90-45dd-bba9-65c3a5507d66-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be4dd34e-df90-45dd-bba9-65c3a5507d66" (UID: "be4dd34e-df90-45dd-bba9-65c3a5507d66"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:56:39.967050 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.967028    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4dd34e-df90-45dd-bba9-65c3a5507d66-kube-api-access-kvmrs" (OuterVolumeSpecName: "kube-api-access-kvmrs") pod "be4dd34e-df90-45dd-bba9-65c3a5507d66" (UID: "be4dd34e-df90-45dd-bba9-65c3a5507d66"). InnerVolumeSpecName "kube-api-access-kvmrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:56:39.980521 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.980504    2557 scope.go:117] "RemoveContainer" containerID="72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba"
May 20 14:56:39.980767 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:56:39.980741    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba\": container with ID starting with 72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba not found: ID does not exist" containerID="72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba"
May 20 14:56:39.980863 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.980779    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba"} err="failed to get container status \"72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba\": rpc error: code = NotFound desc = could not find container \"72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba\": container with ID starting with 72ca5d314d0f98ffb4759049f39bd2ae6fef7635262610e177de0fceba487dba not found: ID does not exist"
May 20 14:56:39.980863 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.980801    2557 scope.go:117] "RemoveContainer" containerID="6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769"
May 20 14:56:39.981064 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:56:39.981043    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769\": container with ID starting with 6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769 not found: ID does not exist" containerID="6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769"
May 20 14:56:39.981105 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.981072    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769"} err="failed to get container status \"6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769\": rpc error: code = NotFound desc = could not find container \"6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769\": container with ID starting with 6b9dbe58bd4f2314bfc670adccf9f4b6dccfdc5fe1410d210449ae4228870769 not found: ID does not exist"
May 20 14:56:39.981105 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.981088    2557 scope.go:117] "RemoveContainer" containerID="a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1"
May 20 14:56:39.981307 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:56:39.981289    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1\": container with ID starting with a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1 not found: ID does not exist" containerID="a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1"
May 20 14:56:39.981342 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:39.981313    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1"} err="failed to get container status \"a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1\": rpc error: code = NotFound desc = could not find container \"a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1\": container with ID starting with a04d3660ea68e8dd8797345d6e3dcf89cc80e3ff51383ef37bcbbc7e734ee4f1 not found: ID does not exist"
May 20 14:56:40.066292 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:40.066266    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be4dd34e-df90-45dd-bba9-65c3a5507d66-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:56:40.066292 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:40.066290    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvmrs\" (UniqueName: \"kubernetes.io/projected/be4dd34e-df90-45dd-bba9-65c3a5507d66-kube-api-access-kvmrs\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:56:40.273907 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:40.273872    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"]
May 20 14:56:40.278304 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:40.278283    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-75fb8cbf6b-xk87v"]
May 20 14:56:40.405738 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:40.405713    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" path="/var/lib/kubelet/pods/be4dd34e-df90-45dd-bba9-65c3a5507d66/volumes"
May 20 14:56:44.970089 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:44.970062    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv_c1c1e5a2-53c8-4854-bb20-42cb6a219182/storage-initializer/0.log"
May 20 14:56:44.970494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:44.970106    2557 generic.go:358] "Generic (PLEG): container finished" podID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" containerID="2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306" exitCode=1
May 20 14:56:44.970494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:44.970203    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv" event={"ID":"c1c1e5a2-53c8-4854-bb20-42cb6a219182","Type":"ContainerDied","Data":"2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306"}
May 20 14:56:45.975238 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:45.975208    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv_c1c1e5a2-53c8-4854-bb20-42cb6a219182/storage-initializer/0.log"
May 20 14:56:45.975622 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:45.975259    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv" event={"ID":"c1c1e5a2-53c8-4854-bb20-42cb6a219182","Type":"ContainerStarted","Data":"531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f"}
May 20 14:56:47.272163 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:47.272131    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"]
May 20 14:56:47.272539 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:47.272380    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv" podUID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" containerName="storage-initializer" containerID="cri-o://531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f" gracePeriod=30
May 20 14:56:48.338701 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.338666    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"]
May 20 14:56:48.339087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.339052    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="storage-initializer"
May 20 14:56:48.339087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.339064    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="storage-initializer"
May 20 14:56:48.339087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.339076    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kube-rbac-proxy"
May 20 14:56:48.339087 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.339084    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kube-rbac-proxy"
May 20 14:56:48.339214 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.339109    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container"
May 20 14:56:48.339214 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.339116    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container"
May 20 14:56:48.339214 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.339184    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kube-rbac-proxy"
May 20 14:56:48.339214 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.339192    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="be4dd34e-df90-45dd-bba9-65c3a5507d66" containerName="kserve-container"
May 20 14:56:48.342420 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.342403    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.347548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.347524    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
May 20 14:56:48.347548 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.347539    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 20 14:56:48.347717 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.347588    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
May 20 14:56:48.357405 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.357382    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"]
May 20 14:56:48.427448 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.427421    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fff05c5-4f0a-402b-8164-50767d508ed0-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.427535 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.427455    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.427535 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.427517    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.427625 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.427590    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7bms\" (UniqueName: \"kubernetes.io/projected/9fff05c5-4f0a-402b-8164-50767d508ed0-kube-api-access-j7bms\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.427625 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.427613    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fff05c5-4f0a-402b-8164-50767d508ed0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.528385 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.528358    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.528477 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.528414    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j7bms\" (UniqueName: \"kubernetes.io/projected/9fff05c5-4f0a-402b-8164-50767d508ed0-kube-api-access-j7bms\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.528477 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.528437    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fff05c5-4f0a-402b-8164-50767d508ed0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.528477 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.528460    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fff05c5-4f0a-402b-8164-50767d508ed0-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.528622 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.528480    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.528622 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:56:48.528581    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
May 20 14:56:48.528703 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:56:48.528659    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fff05c5-4f0a-402b-8164-50767d508ed0-proxy-tls podName:9fff05c5-4f0a-402b-8164-50767d508ed0 nodeName:}" failed. No retries permitted until 2026-05-20 14:56:49.028641426 +0000 UTC m=+3989.166195623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9fff05c5-4f0a-402b-8164-50767d508ed0-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" (UID: "9fff05c5-4f0a-402b-8164-50767d508ed0") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
May 20 14:56:48.528924 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.528900    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fff05c5-4f0a-402b-8164-50767d508ed0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.529103 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.529084    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.529169 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.529153    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:48.540216 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:48.540195    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7bms\" (UniqueName: \"kubernetes.io/projected/9fff05c5-4f0a-402b-8164-50767d508ed0-kube-api-access-j7bms\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:49.032317 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:49.032290    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fff05c5-4f0a-402b-8164-50767d508ed0-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:49.034632 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:49.034612    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fff05c5-4f0a-402b-8164-50767d508ed0-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:49.253108 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:49.253077    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:49.389901 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:49.389815    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"]
May 20 14:56:49.392145 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:56:49.392118    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fff05c5_4f0a_402b_8164_50767d508ed0.slice/crio-59c04a1d69f232ecc0c207cc52e87a77f27d4c2246c0a1bec07d2b08eea68a52 WatchSource:0}: Error finding container 59c04a1d69f232ecc0c207cc52e87a77f27d4c2246c0a1bec07d2b08eea68a52: Status 404 returned error can't find the container with id 59c04a1d69f232ecc0c207cc52e87a77f27d4c2246c0a1bec07d2b08eea68a52
May 20 14:56:49.993398 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:49.993361    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" event={"ID":"9fff05c5-4f0a-402b-8164-50767d508ed0","Type":"ContainerStarted","Data":"2da901354b86e452e9beae0dd255af2d1f71da611d55c7885b0f5e83d170937c"}
May 20 14:56:49.993398 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:49.993397    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" event={"ID":"9fff05c5-4f0a-402b-8164-50767d508ed0","Type":"ContainerStarted","Data":"59c04a1d69f232ecc0c207cc52e87a77f27d4c2246c0a1bec07d2b08eea68a52"}
May 20 14:56:50.998369 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:50.998333    2557 generic.go:358] "Generic (PLEG): container finished" podID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerID="2da901354b86e452e9beae0dd255af2d1f71da611d55c7885b0f5e83d170937c" exitCode=0
May 20 14:56:50.998738 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:50.998374    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" event={"ID":"9fff05c5-4f0a-402b-8164-50767d508ed0","Type":"ContainerDied","Data":"2da901354b86e452e9beae0dd255af2d1f71da611d55c7885b0f5e83d170937c"}
May 20 14:56:51.321946 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.321927    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv_c1c1e5a2-53c8-4854-bb20-42cb6a219182/storage-initializer/1.log"
May 20 14:56:51.322299 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.322284    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv_c1c1e5a2-53c8-4854-bb20-42cb6a219182/storage-initializer/0.log"
May 20 14:56:51.322371 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.322351    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:51.449517 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.449483    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c1e5a2-53c8-4854-bb20-42cb6a219182-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") "
May 20 14:56:51.449701 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.449540    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kserve-provision-location\") pod \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") "
May 20 14:56:51.449701 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.449606    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qjlv\" (UniqueName: \"kubernetes.io/projected/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kube-api-access-2qjlv\") pod \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") "
May 20 14:56:51.449701 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.449634    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c1e5a2-53c8-4854-bb20-42cb6a219182-proxy-tls\") pod \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\" (UID: \"c1c1e5a2-53c8-4854-bb20-42cb6a219182\") "
May 20 14:56:51.449893 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.449831    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1c1e5a2-53c8-4854-bb20-42cb6a219182" (UID: "c1c1e5a2-53c8-4854-bb20-42cb6a219182"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:56:51.449893 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.449873    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c1e5a2-53c8-4854-bb20-42cb6a219182-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "c1c1e5a2-53c8-4854-bb20-42cb6a219182" (UID: "c1c1e5a2-53c8-4854-bb20-42cb6a219182"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:56:51.451616 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.451593    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kube-api-access-2qjlv" (OuterVolumeSpecName: "kube-api-access-2qjlv") pod "c1c1e5a2-53c8-4854-bb20-42cb6a219182" (UID: "c1c1e5a2-53c8-4854-bb20-42cb6a219182"). InnerVolumeSpecName "kube-api-access-2qjlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:56:51.451670 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.451644    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c1e5a2-53c8-4854-bb20-42cb6a219182-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1c1e5a2-53c8-4854-bb20-42cb6a219182" (UID: "c1c1e5a2-53c8-4854-bb20-42cb6a219182"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:56:51.551051 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.551026    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c1e5a2-53c8-4854-bb20-42cb6a219182-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:56:51.551051 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.551049    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:56:51.551255 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.551062    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qjlv\" (UniqueName: \"kubernetes.io/projected/c1c1e5a2-53c8-4854-bb20-42cb6a219182-kube-api-access-2qjlv\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:56:51.551255 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:51.551078    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c1e5a2-53c8-4854-bb20-42cb6a219182-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:56:52.003974 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.003936    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" event={"ID":"9fff05c5-4f0a-402b-8164-50767d508ed0","Type":"ContainerStarted","Data":"cc7ed197a13d6f16359f9ccdcaf220ba1c390ac632b2aaf6b8db491212af10e3"}
May 20 14:56:52.004407 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.003985    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" event={"ID":"9fff05c5-4f0a-402b-8164-50767d508ed0","Type":"ContainerStarted","Data":"173cb0907695f6e0c4777b12fad6193dc0495fbdbf93375003f7796579722d9a"}
May 20 14:56:52.004407 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.004120    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:52.005128 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.005109    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv_c1c1e5a2-53c8-4854-bb20-42cb6a219182/storage-initializer/1.log"
May 20 14:56:52.005465 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.005450    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv_c1c1e5a2-53c8-4854-bb20-42cb6a219182/storage-initializer/0.log"
May 20 14:56:52.005535 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.005485    2557 generic.go:358] "Generic (PLEG): container finished" podID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" containerID="531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f" exitCode=1
May 20 14:56:52.005535 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.005514    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv" event={"ID":"c1c1e5a2-53c8-4854-bb20-42cb6a219182","Type":"ContainerDied","Data":"531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f"}
May 20 14:56:52.005653 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.005558    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"
May 20 14:56:52.005653 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.005571    2557 scope.go:117] "RemoveContainer" containerID="531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f"
May 20 14:56:52.005735 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.005557    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv" event={"ID":"c1c1e5a2-53c8-4854-bb20-42cb6a219182","Type":"ContainerDied","Data":"638ce94dd8b5519e8c3c486c02164a8afe2a64c392a636ba26c8dfd80626662c"}
May 20 14:56:52.016368 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.016347    2557 scope.go:117] "RemoveContainer" containerID="2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306"
May 20 14:56:52.024183 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.024167    2557 scope.go:117] "RemoveContainer" containerID="531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f"
May 20 14:56:52.024421 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:56:52.024405    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f\": container with ID starting with 531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f not found: ID does not exist" containerID="531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f"
May 20 14:56:52.024490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.024432    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f"} err="failed to get container status \"531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f\": rpc error: code = NotFound desc = could not find container \"531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f\": container with ID starting with 531f34a361be15e3c40f6bdcc74a1434737e135c95b33275a8d892debb099b3f not found: ID does not exist"
May 20 14:56:52.024490 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.024455    2557 scope.go:117] "RemoveContainer" containerID="2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306"
May 20 14:56:52.024733 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:56:52.024680    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306\": container with ID starting with 2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306 not found: ID does not exist" containerID="2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306"
May 20 14:56:52.024733 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.024712    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306"} err="failed to get container status \"2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306\": rpc error: code = NotFound desc = could not find container \"2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306\": container with ID starting with 2256bbb1c5c571f46f8fd3b8db3a78172be96653e92acae1f0695af151dd8306 not found: ID does not exist"
May 20 14:56:52.027399 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.027354    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podStartSLOduration=4.027340825 podStartE2EDuration="4.027340825s" podCreationTimestamp="2026-05-20 14:56:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:56:52.025072135 +0000 UTC m=+3992.162626365" watchObservedRunningTime="2026-05-20 14:56:52.027340825 +0000 UTC m=+3992.164895045"
May 20 14:56:52.058153 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.058127    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"]
May 20 14:56:52.062840 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.062819    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d65c47c8d-nfnfv"]
May 20 14:56:52.404796 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:52.404763    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" path="/var/lib/kubelet/pods/c1c1e5a2-53c8-4854-bb20-42cb6a219182/volumes"
May 20 14:56:53.010270 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:53.010239    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:53.011348 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:53.011320    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 20 14:56:54.013916 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:54.013882    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 20 14:56:59.019665 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:59.019633    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:56:59.020189 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:56:59.020121    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 20 14:57:09.020331 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:57:09.020288    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 20 14:57:19.020755 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:57:19.020720    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 20 14:57:29.020561 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:57:29.020518    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 20 14:57:39.020759 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:57:39.020674    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 20 14:57:49.020559 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:57:49.020522    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 20 14:57:59.020936 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:57:59.020902    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:58:08.370602 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:08.370558    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"]
May 20 14:58:08.371112 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:08.370885    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" containerID="cri-o://173cb0907695f6e0c4777b12fad6193dc0495fbdbf93375003f7796579722d9a" gracePeriod=30
May 20 14:58:08.371112 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:08.370921    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kube-rbac-proxy" containerID="cri-o://cc7ed197a13d6f16359f9ccdcaf220ba1c390ac632b2aaf6b8db491212af10e3" gracePeriod=30
May 20 14:58:09.014273 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.014235    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
May 20 14:58:09.020705 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.020674    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 20 14:58:09.265093 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.265017    2557 generic.go:358] "Generic (PLEG): container finished" podID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerID="cc7ed197a13d6f16359f9ccdcaf220ba1c390ac632b2aaf6b8db491212af10e3" exitCode=2
May 20 14:58:09.265093 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.265056    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" event={"ID":"9fff05c5-4f0a-402b-8164-50767d508ed0","Type":"ContainerDied","Data":"cc7ed197a13d6f16359f9ccdcaf220ba1c390ac632b2aaf6b8db491212af10e3"}
May 20 14:58:09.468391 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.468367    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"]
May 20 14:58:09.468810 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.468793    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" containerName="storage-initializer"
May 20 14:58:09.468885 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.468813    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" containerName="storage-initializer"
May 20 14:58:09.468885 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.468822    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" containerName="storage-initializer"
May 20 14:58:09.468885 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.468828    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" containerName="storage-initializer"
May 20 14:58:09.468995 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.468920    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" containerName="storage-initializer"
May 20 14:58:09.469041 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.469030    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1c1e5a2-53c8-4854-bb20-42cb6a219182" containerName="storage-initializer"
May 20 14:58:09.472108 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.472093    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.474673 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.474646    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
May 20 14:58:09.474773 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.474707    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
May 20 14:58:09.483475 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.483453    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"]
May 20 14:58:09.524807 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.524751    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92647e05-7c35-4dd9-9255-b142f069e0ca-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.524807 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.524798    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t4vv\" (UniqueName: \"kubernetes.io/projected/92647e05-7c35-4dd9-9255-b142f069e0ca-kube-api-access-9t4vv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.524965 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.524822    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92647e05-7c35-4dd9-9255-b142f069e0ca-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.524965 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.524936    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92647e05-7c35-4dd9-9255-b142f069e0ca-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.625415 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.625391    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9t4vv\" (UniqueName: \"kubernetes.io/projected/92647e05-7c35-4dd9-9255-b142f069e0ca-kube-api-access-9t4vv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.625513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.625427    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92647e05-7c35-4dd9-9255-b142f069e0ca-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.625513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.625463    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92647e05-7c35-4dd9-9255-b142f069e0ca-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.625513 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.625501    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92647e05-7c35-4dd9-9255-b142f069e0ca-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.625933 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.625912    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92647e05-7c35-4dd9-9255-b142f069e0ca-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.626090 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.626069    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92647e05-7c35-4dd9-9255-b142f069e0ca-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.627943 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.627924    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92647e05-7c35-4dd9-9255-b142f069e0ca-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.634678 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.634651    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t4vv\" (UniqueName: \"kubernetes.io/projected/92647e05-7c35-4dd9-9255-b142f069e0ca-kube-api-access-9t4vv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.782498 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.782442    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:09.926662 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:58:09.926619    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92647e05_7c35_4dd9_9255_b142f069e0ca.slice/crio-7827538c539d42a4616ec95740b5cc9528eee66e75e3ecd37d26874b1271d544 WatchSource:0}: Error finding container 7827538c539d42a4616ec95740b5cc9528eee66e75e3ecd37d26874b1271d544: Status 404 returned error can't find the container with id 7827538c539d42a4616ec95740b5cc9528eee66e75e3ecd37d26874b1271d544
May 20 14:58:09.928069 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.928045    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"]
May 20 14:58:09.928686 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:09.928670    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:58:10.270062 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:10.270024    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5" event={"ID":"92647e05-7c35-4dd9-9255-b142f069e0ca","Type":"ContainerStarted","Data":"121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511"}
May 20 14:58:10.270308 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:10.270067    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5" event={"ID":"92647e05-7c35-4dd9-9255-b142f069e0ca","Type":"ContainerStarted","Data":"7827538c539d42a4616ec95740b5cc9528eee66e75e3ecd37d26874b1271d544"}
May 20 14:58:14.014431 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:14.014391    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
May 20 14:58:15.287469 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.287440    2557 generic.go:358] "Generic (PLEG): container finished" podID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerID="173cb0907695f6e0c4777b12fad6193dc0495fbdbf93375003f7796579722d9a" exitCode=0
May 20 14:58:15.287805 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.287515    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" event={"ID":"9fff05c5-4f0a-402b-8164-50767d508ed0","Type":"ContainerDied","Data":"173cb0907695f6e0c4777b12fad6193dc0495fbdbf93375003f7796579722d9a"}
May 20 14:58:15.314926 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.314908    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:58:15.371494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.371436    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7bms\" (UniqueName: \"kubernetes.io/projected/9fff05c5-4f0a-402b-8164-50767d508ed0-kube-api-access-j7bms\") pod \"9fff05c5-4f0a-402b-8164-50767d508ed0\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") "
May 20 14:58:15.371494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.371474    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"9fff05c5-4f0a-402b-8164-50767d508ed0\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") "
May 20 14:58:15.371635 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.371514    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-cabundle-cert\") pod \"9fff05c5-4f0a-402b-8164-50767d508ed0\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") "
May 20 14:58:15.371635 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.371582    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fff05c5-4f0a-402b-8164-50767d508ed0-proxy-tls\") pod \"9fff05c5-4f0a-402b-8164-50767d508ed0\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") "
May 20 14:58:15.371776 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.371662    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fff05c5-4f0a-402b-8164-50767d508ed0-kserve-provision-location\") pod \"9fff05c5-4f0a-402b-8164-50767d508ed0\" (UID: \"9fff05c5-4f0a-402b-8164-50767d508ed0\") "
May 20 14:58:15.372056 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.372029    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fff05c5-4f0a-402b-8164-50767d508ed0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fff05c5-4f0a-402b-8164-50767d508ed0" (UID: "9fff05c5-4f0a-402b-8164-50767d508ed0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:58:15.372056 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.372031    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9fff05c5-4f0a-402b-8164-50767d508ed0" (UID: "9fff05c5-4f0a-402b-8164-50767d508ed0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:58:15.372190 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.372037    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "9fff05c5-4f0a-402b-8164-50767d508ed0" (UID: "9fff05c5-4f0a-402b-8164-50767d508ed0"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:58:15.373616 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.373586    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fff05c5-4f0a-402b-8164-50767d508ed0-kube-api-access-j7bms" (OuterVolumeSpecName: "kube-api-access-j7bms") pod "9fff05c5-4f0a-402b-8164-50767d508ed0" (UID: "9fff05c5-4f0a-402b-8164-50767d508ed0"). InnerVolumeSpecName "kube-api-access-j7bms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:58:15.373616 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.373599    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fff05c5-4f0a-402b-8164-50767d508ed0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fff05c5-4f0a-402b-8164-50767d508ed0" (UID: "9fff05c5-4f0a-402b-8164-50767d508ed0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:58:15.472279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.472248    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fff05c5-4f0a-402b-8164-50767d508ed0-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:15.472279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.472279    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7bms\" (UniqueName: \"kubernetes.io/projected/9fff05c5-4f0a-402b-8164-50767d508ed0-kube-api-access-j7bms\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:15.472494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.472295    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:15.472494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.472310    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fff05c5-4f0a-402b-8164-50767d508ed0-cabundle-cert\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:15.472494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:15.472322    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fff05c5-4f0a-402b-8164-50767d508ed0-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:16.292275 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.292246    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2" event={"ID":"9fff05c5-4f0a-402b-8164-50767d508ed0","Type":"ContainerDied","Data":"59c04a1d69f232ecc0c207cc52e87a77f27d4c2246c0a1bec07d2b08eea68a52"}
May 20 14:58:16.292661 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.292292    2557 scope.go:117] "RemoveContainer" containerID="cc7ed197a13d6f16359f9ccdcaf220ba1c390ac632b2aaf6b8db491212af10e3"
May 20 14:58:16.292661 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.292304    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"
May 20 14:58:16.293920 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.293904    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5_92647e05-7c35-4dd9-9255-b142f069e0ca/storage-initializer/0.log"
May 20 14:58:16.294011 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.293939    2557 generic.go:358] "Generic (PLEG): container finished" podID="92647e05-7c35-4dd9-9255-b142f069e0ca" containerID="121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511" exitCode=1
May 20 14:58:16.294011 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.293987    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5" event={"ID":"92647e05-7c35-4dd9-9255-b142f069e0ca","Type":"ContainerDied","Data":"121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511"}
May 20 14:58:16.301676 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.301656    2557 scope.go:117] "RemoveContainer" containerID="173cb0907695f6e0c4777b12fad6193dc0495fbdbf93375003f7796579722d9a"
May 20 14:58:16.308847 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.308829    2557 scope.go:117] "RemoveContainer" containerID="2da901354b86e452e9beae0dd255af2d1f71da611d55c7885b0f5e83d170937c"
May 20 14:58:16.337251 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.337232    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"]
May 20 14:58:16.342755 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.342729    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5b5674f5df-gx2z2"]
May 20 14:58:16.405562 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:16.405523    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" path="/var/lib/kubelet/pods/9fff05c5-4f0a-402b-8164-50767d508ed0/volumes"
May 20 14:58:17.299520 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:17.299491    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5_92647e05-7c35-4dd9-9255-b142f069e0ca/storage-initializer/0.log"
May 20 14:58:17.299949 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:17.299589    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5" event={"ID":"92647e05-7c35-4dd9-9255-b142f069e0ca","Type":"ContainerStarted","Data":"17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56"}
May 20 14:58:19.456209 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:19.456179    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"]
May 20 14:58:19.456631 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:19.456420    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5" podUID="92647e05-7c35-4dd9-9255-b142f069e0ca" containerName="storage-initializer" containerID="cri-o://17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56" gracePeriod=30
May 20 14:58:21.151930 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.151894    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tbrjs/must-gather-sv95j"]
May 20 14:58:21.152285 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.152256    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container"
May 20 14:58:21.152285 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.152270    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container"
May 20 14:58:21.152361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.152289    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kube-rbac-proxy"
May 20 14:58:21.152361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.152296    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kube-rbac-proxy"
May 20 14:58:21.152361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.152318    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="storage-initializer"
May 20 14:58:21.152361 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.152325    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="storage-initializer"
May 20 14:58:21.152488 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.152396    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kube-rbac-proxy"
May 20 14:58:21.152488 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.152409    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="9fff05c5-4f0a-402b-8164-50767d508ed0" containerName="kserve-container"
May 20 14:58:21.157335 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.157298    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:21.160433 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.160409    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tbrjs\"/\"kube-root-ca.crt\""
May 20 14:58:21.161096 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.161075    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tbrjs\"/\"openshift-service-ca.crt\""
May 20 14:58:21.167902 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.167883    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tbrjs/must-gather-sv95j"]
May 20 14:58:21.220089 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.220061    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hstk\" (UniqueName: \"kubernetes.io/projected/9de8c541-e168-42ff-b36b-e27798c8e59e-kube-api-access-7hstk\") pod \"must-gather-sv95j\" (UID: \"9de8c541-e168-42ff-b36b-e27798c8e59e\") " pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:21.220209 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.220114    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9de8c541-e168-42ff-b36b-e27798c8e59e-must-gather-output\") pod \"must-gather-sv95j\" (UID: \"9de8c541-e168-42ff-b36b-e27798c8e59e\") " pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:21.320571 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.320547    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7hstk\" (UniqueName: \"kubernetes.io/projected/9de8c541-e168-42ff-b36b-e27798c8e59e-kube-api-access-7hstk\") pod \"must-gather-sv95j\" (UID: \"9de8c541-e168-42ff-b36b-e27798c8e59e\") " pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:21.320695 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.320584    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9de8c541-e168-42ff-b36b-e27798c8e59e-must-gather-output\") pod \"must-gather-sv95j\" (UID: \"9de8c541-e168-42ff-b36b-e27798c8e59e\") " pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:21.320839 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.320824    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9de8c541-e168-42ff-b36b-e27798c8e59e-must-gather-output\") pod \"must-gather-sv95j\" (UID: \"9de8c541-e168-42ff-b36b-e27798c8e59e\") " pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:21.331491 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.331468    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hstk\" (UniqueName: \"kubernetes.io/projected/9de8c541-e168-42ff-b36b-e27798c8e59e-kube-api-access-7hstk\") pod \"must-gather-sv95j\" (UID: \"9de8c541-e168-42ff-b36b-e27798c8e59e\") " pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:21.477171 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.477121    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:21.621941 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:21.621765    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tbrjs/must-gather-sv95j"]
May 20 14:58:22.324583 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.324543    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tbrjs/must-gather-sv95j" event={"ID":"9de8c541-e168-42ff-b36b-e27798c8e59e","Type":"ContainerStarted","Data":"415e98add30a7f9e8cdbd87b1423f76d2cbaa4875af07a4c6077489bc51f8d04"}
May 20 14:58:22.589928 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.589793    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5_92647e05-7c35-4dd9-9255-b142f069e0ca/storage-initializer/1.log"
May 20 14:58:22.590727 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.590237    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5_92647e05-7c35-4dd9-9255-b142f069e0ca/storage-initializer/0.log"
May 20 14:58:22.590727 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.590312    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:22.734418 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.734379    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92647e05-7c35-4dd9-9255-b142f069e0ca-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"92647e05-7c35-4dd9-9255-b142f069e0ca\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") "
May 20 14:58:22.734601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.734469    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92647e05-7c35-4dd9-9255-b142f069e0ca-proxy-tls\") pod \"92647e05-7c35-4dd9-9255-b142f069e0ca\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") "
May 20 14:58:22.734601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.734533    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92647e05-7c35-4dd9-9255-b142f069e0ca-kserve-provision-location\") pod \"92647e05-7c35-4dd9-9255-b142f069e0ca\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") "
May 20 14:58:22.734601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.734561    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t4vv\" (UniqueName: \"kubernetes.io/projected/92647e05-7c35-4dd9-9255-b142f069e0ca-kube-api-access-9t4vv\") pod \"92647e05-7c35-4dd9-9255-b142f069e0ca\" (UID: \"92647e05-7c35-4dd9-9255-b142f069e0ca\") "
May 20 14:58:22.735395 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.735367    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92647e05-7c35-4dd9-9255-b142f069e0ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92647e05-7c35-4dd9-9255-b142f069e0ca" (UID: "92647e05-7c35-4dd9-9255-b142f069e0ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:58:22.735538 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.735511    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92647e05-7c35-4dd9-9255-b142f069e0ca-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "92647e05-7c35-4dd9-9255-b142f069e0ca" (UID: "92647e05-7c35-4dd9-9255-b142f069e0ca"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:58:22.741540 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.741512    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92647e05-7c35-4dd9-9255-b142f069e0ca-kube-api-access-9t4vv" (OuterVolumeSpecName: "kube-api-access-9t4vv") pod "92647e05-7c35-4dd9-9255-b142f069e0ca" (UID: "92647e05-7c35-4dd9-9255-b142f069e0ca"). InnerVolumeSpecName "kube-api-access-9t4vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:58:22.743771 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.743744    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92647e05-7c35-4dd9-9255-b142f069e0ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92647e05-7c35-4dd9-9255-b142f069e0ca" (UID: "92647e05-7c35-4dd9-9255-b142f069e0ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:58:22.836311 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.836252    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92647e05-7c35-4dd9-9255-b142f069e0ca-proxy-tls\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:22.836311 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.836290    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92647e05-7c35-4dd9-9255-b142f069e0ca-kserve-provision-location\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:22.836311 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.836307    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t4vv\" (UniqueName: \"kubernetes.io/projected/92647e05-7c35-4dd9-9255-b142f069e0ca-kube-api-access-9t4vv\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:22.836599 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:22.836325    2557 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92647e05-7c35-4dd9-9255-b142f069e0ca-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:23.332626 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.332594    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5_92647e05-7c35-4dd9-9255-b142f069e0ca/storage-initializer/1.log"
May 20 14:58:23.333143 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.333088    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5_92647e05-7c35-4dd9-9255-b142f069e0ca/storage-initializer/0.log"
May 20 14:58:23.333143 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.333125    2557 generic.go:358] "Generic (PLEG): container finished" podID="92647e05-7c35-4dd9-9255-b142f069e0ca" containerID="17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56" exitCode=1
May 20 14:58:23.333362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.333222    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"
May 20 14:58:23.333362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.333262    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5" event={"ID":"92647e05-7c35-4dd9-9255-b142f069e0ca","Type":"ContainerDied","Data":"17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56"}
May 20 14:58:23.333362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.333294    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5" event={"ID":"92647e05-7c35-4dd9-9255-b142f069e0ca","Type":"ContainerDied","Data":"7827538c539d42a4616ec95740b5cc9528eee66e75e3ecd37d26874b1271d544"}
May 20 14:58:23.333362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.333314    2557 scope.go:117] "RemoveContainer" containerID="17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56"
May 20 14:58:23.349728 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.349712    2557 scope.go:117] "RemoveContainer" containerID="121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511"
May 20 14:58:23.363707 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.363665    2557 scope.go:117] "RemoveContainer" containerID="17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56"
May 20 14:58:23.364014 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:58:23.363987    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56\": container with ID starting with 17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56 not found: ID does not exist" containerID="17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56"
May 20 14:58:23.364099 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.364027    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56"} err="failed to get container status \"17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56\": rpc error: code = NotFound desc = could not find container \"17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56\": container with ID starting with 17cddca0763df9de0ab59b92d7d7c927dbc3e4de3a91c6606a52f477dbe6ff56 not found: ID does not exist"
May 20 14:58:23.364099 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.364052    2557 scope.go:117] "RemoveContainer" containerID="121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511"
May 20 14:58:23.364349 ip-10-0-139-248 kubenswrapper[2557]: E0520 14:58:23.364308    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511\": container with ID starting with 121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511 not found: ID does not exist" containerID="121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511"
May 20 14:58:23.364419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.364336    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511"} err="failed to get container status \"121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511\": rpc error: code = NotFound desc = could not find container \"121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511\": container with ID starting with 121ca5a9bbf29e03c3331a2029aa6a98daa1d27f262a7604752e497c9f427511 not found: ID does not exist"
May 20 14:58:23.391480 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.391451    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"]
May 20 14:58:23.393157 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:23.393135    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-756b846d59-tm4l5"]
May 20 14:58:24.407074 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:24.407039    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92647e05-7c35-4dd9-9255-b142f069e0ca" path="/var/lib/kubelet/pods/92647e05-7c35-4dd9-9255-b142f069e0ca/volumes"
May 20 14:58:28.355282 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:28.355245    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tbrjs/must-gather-sv95j" event={"ID":"9de8c541-e168-42ff-b36b-e27798c8e59e","Type":"ContainerStarted","Data":"de5ce6f3d46d5ff57b7e9129c34d74050119f2641a3ba9dd35d9b0064d1797d3"}
May 20 14:58:28.355282 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:28.355281    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tbrjs/must-gather-sv95j" event={"ID":"9de8c541-e168-42ff-b36b-e27798c8e59e","Type":"ContainerStarted","Data":"e201df6e25ebad0e19ab46884a62c4de1c4d73612c011f03846c453c49d36292"}
May 20 14:58:28.384708 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:28.384655    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-tbrjs/must-gather-sv95j" podStartSLOduration=1.285517274 podStartE2EDuration="7.384636568s" podCreationTimestamp="2026-05-20 14:58:21 +0000 UTC" firstStartedPulling="2026-05-20 14:58:21.626097175 +0000 UTC m=+4081.763651373" lastFinishedPulling="2026-05-20 14:58:27.725216457 +0000 UTC m=+4087.862770667" observedRunningTime="2026-05-20 14:58:28.383189889 +0000 UTC m=+4088.520744107" watchObservedRunningTime="2026-05-20 14:58:28.384636568 +0000 UTC m=+4088.522190788"
May 20 14:58:49.429037 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:49.428998    2557 generic.go:358] "Generic (PLEG): container finished" podID="9de8c541-e168-42ff-b36b-e27798c8e59e" containerID="e201df6e25ebad0e19ab46884a62c4de1c4d73612c011f03846c453c49d36292" exitCode=0
May 20 14:58:49.429456 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:49.429068    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tbrjs/must-gather-sv95j" event={"ID":"9de8c541-e168-42ff-b36b-e27798c8e59e","Type":"ContainerDied","Data":"e201df6e25ebad0e19ab46884a62c4de1c4d73612c011f03846c453c49d36292"}
May 20 14:58:49.429456 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:49.429371    2557 scope.go:117] "RemoveContainer" containerID="e201df6e25ebad0e19ab46884a62c4de1c4d73612c011f03846c453c49d36292"
May 20 14:58:49.732571 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:49.732482    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tbrjs_must-gather-sv95j_9de8c541-e168-42ff-b36b-e27798c8e59e/gather/0.log"
May 20 14:58:53.400462 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:53.400424    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tcj8j_0e804929-2290-4e4c-b3d6-c9ea04e60e40/global-pull-secret-syncer/0.log"
May 20 14:58:53.550721 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:53.550682    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wc5n2_18cd97d3-32d6-4d7a-9c27-5d014aff0dbb/konnectivity-agent/0.log"
May 20 14:58:53.636626 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:53.636597    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-248.ec2.internal_c7bb2fc9c4d251ff13095ab5d1d57128/haproxy/0.log"
May 20 14:58:55.292744 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.292708    2557 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-tbrjs/must-gather-sv95j"]
May 20 14:58:55.293173 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.292937    2557 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-tbrjs/must-gather-sv95j" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" containerName="copy" containerID="cri-o://de5ce6f3d46d5ff57b7e9129c34d74050119f2641a3ba9dd35d9b0064d1797d3" gracePeriod=2
May 20 14:58:55.295994 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.295964    2557 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-tbrjs/must-gather-sv95j"]
May 20 14:58:55.297656 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.297627    2557 status_manager.go:895] "Failed to get status for pod" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" pod="openshift-must-gather-tbrjs/must-gather-sv95j" err="pods \"must-gather-sv95j\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-tbrjs\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object"
May 20 14:58:55.450832 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.450810    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tbrjs_must-gather-sv95j_9de8c541-e168-42ff-b36b-e27798c8e59e/copy/0.log"
May 20 14:58:55.451198 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.451174    2557 generic.go:358] "Generic (PLEG): container finished" podID="9de8c541-e168-42ff-b36b-e27798c8e59e" containerID="de5ce6f3d46d5ff57b7e9129c34d74050119f2641a3ba9dd35d9b0064d1797d3" exitCode=143
May 20 14:58:55.518613 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.518597    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tbrjs_must-gather-sv95j_9de8c541-e168-42ff-b36b-e27798c8e59e/copy/0.log"
May 20 14:58:55.518973 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.518957    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:55.522204 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.522176    2557 status_manager.go:895] "Failed to get status for pod" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" pod="openshift-must-gather-tbrjs/must-gather-sv95j" err="pods \"must-gather-sv95j\" is forbidden: User \"system:node:ip-10-0-139-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-tbrjs\": no relationship found between node 'ip-10-0-139-248.ec2.internal' and this object"
May 20 14:58:55.612709 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.612687    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hstk\" (UniqueName: \"kubernetes.io/projected/9de8c541-e168-42ff-b36b-e27798c8e59e-kube-api-access-7hstk\") pod \"9de8c541-e168-42ff-b36b-e27798c8e59e\" (UID: \"9de8c541-e168-42ff-b36b-e27798c8e59e\") "
May 20 14:58:55.612823 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.612735    2557 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9de8c541-e168-42ff-b36b-e27798c8e59e-must-gather-output\") pod \"9de8c541-e168-42ff-b36b-e27798c8e59e\" (UID: \"9de8c541-e168-42ff-b36b-e27798c8e59e\") "
May 20 14:58:55.614254 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.614229    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de8c541-e168-42ff-b36b-e27798c8e59e-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9de8c541-e168-42ff-b36b-e27798c8e59e" (UID: "9de8c541-e168-42ff-b36b-e27798c8e59e"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:58:55.614844 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.614817    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de8c541-e168-42ff-b36b-e27798c8e59e-kube-api-access-7hstk" (OuterVolumeSpecName: "kube-api-access-7hstk") pod "9de8c541-e168-42ff-b36b-e27798c8e59e" (UID: "9de8c541-e168-42ff-b36b-e27798c8e59e"). InnerVolumeSpecName "kube-api-access-7hstk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:58:55.714013 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.713987    2557 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9de8c541-e168-42ff-b36b-e27798c8e59e-must-gather-output\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:55.714013 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:55.714012    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hstk\" (UniqueName: \"kubernetes.io/projected/9de8c541-e168-42ff-b36b-e27798c8e59e-kube-api-access-7hstk\") on node \"ip-10-0-139-248.ec2.internal\" DevicePath \"\""
May 20 14:58:56.405112 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.405084    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" path="/var/lib/kubelet/pods/9de8c541-e168-42ff-b36b-e27798c8e59e/volumes"
May 20 14:58:56.455060 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.455041    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tbrjs_must-gather-sv95j_9de8c541-e168-42ff-b36b-e27798c8e59e/copy/0.log"
May 20 14:58:56.455508 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.455489    2557 scope.go:117] "RemoveContainer" containerID="de5ce6f3d46d5ff57b7e9129c34d74050119f2641a3ba9dd35d9b0064d1797d3"
May 20 14:58:56.455599 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.455512    2557 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tbrjs/must-gather-sv95j"
May 20 14:58:56.462891 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.462869    2557 scope.go:117] "RemoveContainer" containerID="e201df6e25ebad0e19ab46884a62c4de1c4d73612c011f03846c453c49d36292"
May 20 14:58:56.777484 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.777459    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71936781-21e0-4444-9bc4-c23d7840ae67/alertmanager/0.log"
May 20 14:58:56.802254 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.802230    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71936781-21e0-4444-9bc4-c23d7840ae67/config-reloader/0.log"
May 20 14:58:56.827608 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.827584    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71936781-21e0-4444-9bc4-c23d7840ae67/kube-rbac-proxy-web/0.log"
May 20 14:58:56.855784 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.855747    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71936781-21e0-4444-9bc4-c23d7840ae67/kube-rbac-proxy/0.log"
May 20 14:58:56.902005 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.901987    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71936781-21e0-4444-9bc4-c23d7840ae67/kube-rbac-proxy-metric/0.log"
May 20 14:58:56.932064 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.932012    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71936781-21e0-4444-9bc4-c23d7840ae67/prom-label-proxy/0.log"
May 20 14:58:56.957341 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:56.957319    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_71936781-21e0-4444-9bc4-c23d7840ae67/init-config-reloader/0.log"
May 20 14:58:57.043315 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.043280    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-rvzhf_011b9ea8-4788-46a9-9cb6-d5c6f20b843d/kube-state-metrics/0.log"
May 20 14:58:57.073170 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.073150    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-rvzhf_011b9ea8-4788-46a9-9cb6-d5c6f20b843d/kube-rbac-proxy-main/0.log"
May 20 14:58:57.098197 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.098178    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-rvzhf_011b9ea8-4788-46a9-9cb6-d5c6f20b843d/kube-rbac-proxy-self/0.log"
May 20 14:58:57.338480 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.338458    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sx46z_9bac2ac0-f861-498f-92f6-049402b8f6d6/node-exporter/0.log"
May 20 14:58:57.361960 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.361938    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sx46z_9bac2ac0-f861-498f-92f6-049402b8f6d6/kube-rbac-proxy/0.log"
May 20 14:58:57.387029 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.387012    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sx46z_9bac2ac0-f861-498f-92f6-049402b8f6d6/init-textfile/0.log"
May 20 14:58:57.417458 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.417436    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6sb49_bb528ea5-f9cc-4ff8-91d2-49c5452bf8be/kube-rbac-proxy-main/0.log"
May 20 14:58:57.448546 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.448515    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6sb49_bb528ea5-f9cc-4ff8-91d2-49c5452bf8be/kube-rbac-proxy-self/0.log"
May 20 14:58:57.472356 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.472334    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-6sb49_bb528ea5-f9cc-4ff8-91d2-49c5452bf8be/openshift-state-metrics/0.log"
May 20 14:58:57.518732 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.518706    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70d7745b-dc43-4d35-8506-00a0df10b98b/prometheus/0.log"
May 20 14:58:57.536450 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.536427    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70d7745b-dc43-4d35-8506-00a0df10b98b/config-reloader/0.log"
May 20 14:58:57.562186 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.562167    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70d7745b-dc43-4d35-8506-00a0df10b98b/thanos-sidecar/0.log"
May 20 14:58:57.586437 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.586403    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70d7745b-dc43-4d35-8506-00a0df10b98b/kube-rbac-proxy-web/0.log"
May 20 14:58:57.612685 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.612627    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70d7745b-dc43-4d35-8506-00a0df10b98b/kube-rbac-proxy/0.log"
May 20 14:58:57.636263 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.636247    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70d7745b-dc43-4d35-8506-00a0df10b98b/kube-rbac-proxy-thanos/0.log"
May 20 14:58:57.663201 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.663186    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70d7745b-dc43-4d35-8506-00a0df10b98b/init-config-reloader/0.log"
May 20 14:58:57.854262 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.854240    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dc49d9f-5kmkc_607041de-41d1-4277-b2c4-e6b7f354c9a6/thanos-query/0.log"
May 20 14:58:57.887537 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.887471    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dc49d9f-5kmkc_607041de-41d1-4277-b2c4-e6b7f354c9a6/kube-rbac-proxy-web/0.log"
May 20 14:58:57.916144 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.916119    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dc49d9f-5kmkc_607041de-41d1-4277-b2c4-e6b7f354c9a6/kube-rbac-proxy/0.log"
May 20 14:58:57.956137 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.956117    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dc49d9f-5kmkc_607041de-41d1-4277-b2c4-e6b7f354c9a6/prom-label-proxy/0.log"
May 20 14:58:57.990303 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:57.990271    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dc49d9f-5kmkc_607041de-41d1-4277-b2c4-e6b7f354c9a6/kube-rbac-proxy-rules/0.log"
May 20 14:58:58.025252 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:58.025233    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dc49d9f-5kmkc_607041de-41d1-4277-b2c4-e6b7f354c9a6/kube-rbac-proxy-metrics/0.log"
May 20 14:58:59.525520 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:59.525404    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/2.log"
May 20 14:58:59.529431 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:58:59.529414    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-vgn62_9ed23ead-6ce7-4b98-a8f3-042a5d72979c/console-operator/3.log"
May 20 14:59:00.370959 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.370924    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-bkvlp_02e0e4f9-5851-4545-9e74-6782b356740c/volume-data-source-validator/0.log"
May 20 14:59:00.376466 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.376438    2557 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"]
May 20 14:59:00.376836 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.376819    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92647e05-7c35-4dd9-9255-b142f069e0ca" containerName="storage-initializer"
May 20 14:59:00.376987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.376838    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="92647e05-7c35-4dd9-9255-b142f069e0ca" containerName="storage-initializer"
May 20 14:59:00.376987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.376878    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" containerName="gather"
May 20 14:59:00.376987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.376888    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" containerName="gather"
May 20 14:59:00.376987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.376898    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" containerName="copy"
May 20 14:59:00.376987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.376906    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" containerName="copy"
May 20 14:59:00.376987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.376917    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92647e05-7c35-4dd9-9255-b142f069e0ca" containerName="storage-initializer"
May 20 14:59:00.376987 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.376925    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="92647e05-7c35-4dd9-9255-b142f069e0ca" containerName="storage-initializer"
May 20 14:59:00.377340 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.377036    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="92647e05-7c35-4dd9-9255-b142f069e0ca" containerName="storage-initializer"
May 20 14:59:00.377340 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.377050    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" containerName="copy"
May 20 14:59:00.377340 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.377066    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="92647e05-7c35-4dd9-9255-b142f069e0ca" containerName="storage-initializer"
May 20 14:59:00.377340 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.377077    2557 memory_manager.go:356] "RemoveStaleState removing state" podUID="9de8c541-e168-42ff-b36b-e27798c8e59e" containerName="gather"
May 20 14:59:00.380164 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.380145    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.383958 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.383935    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-674pn\"/\"kube-root-ca.crt\""
May 20 14:59:00.384055 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.383935    2557 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-674pn\"/\"default-dockercfg-lsp5c\""
May 20 14:59:00.385157 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.385137    2557 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-674pn\"/\"openshift-service-ca.crt\""
May 20 14:59:00.388836 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.388817    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"]
May 20 14:59:00.452475 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.452447    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-lib-modules\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.452601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.452526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-podres\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.452601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.452563    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-proc\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.452601 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.452588    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-sys\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.452745 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.452605    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwmb\" (UniqueName: \"kubernetes.io/projected/df6a49b3-8428-482c-9472-8a40b5c69f6c-kube-api-access-gvwmb\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.553212 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.553179    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-proc\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.553212 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.553217    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-sys\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.553662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.553235    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwmb\" (UniqueName: \"kubernetes.io/projected/df6a49b3-8428-482c-9472-8a40b5c69f6c-kube-api-access-gvwmb\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.553662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.553284    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-lib-modules\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.553662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.553293    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-proc\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.553662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.553317    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-sys\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.553662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.553320    2557 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-podres\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.553662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.553401    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-podres\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.553662 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.553431    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df6a49b3-8428-482c-9472-8a40b5c69f6c-lib-modules\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.562419 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.562395    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwmb\" (UniqueName: \"kubernetes.io/projected/df6a49b3-8428-482c-9472-8a40b5c69f6c-kube-api-access-gvwmb\") pod \"perf-node-gather-daemonset-z888g\" (UID: \"df6a49b3-8428-482c-9472-8a40b5c69f6c\") " pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.690041 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.689932    2557 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:00.820906 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:00.820877    2557 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"]
May 20 14:59:00.822715 ip-10-0-139-248 kubenswrapper[2557]: W0520 14:59:00.822688    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddf6a49b3_8428_482c_9472_8a40b5c69f6c.slice/crio-d498318bde0d933e3b258f0d99b88a6a6ca92f45e31d6b8c8055f48777e6a500 WatchSource:0}: Error finding container d498318bde0d933e3b258f0d99b88a6a6ca92f45e31d6b8c8055f48777e6a500: Status 404 returned error can't find the container with id d498318bde0d933e3b258f0d99b88a6a6ca92f45e31d6b8c8055f48777e6a500
May 20 14:59:01.075796 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:01.075774    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8vcdb_ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75/dns/0.log"
May 20 14:59:01.097932 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:01.097907    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8vcdb_ad1c4d6d-dcf4-4d04-8e49-c846e28d1f75/kube-rbac-proxy/0.log"
May 20 14:59:01.213623 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:01.213601    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-58wjx_81d61543-18a2-43a7-b322-54bdaf92f4a9/dns-node-resolver/0.log"
May 20 14:59:01.472136 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:01.472067    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g" event={"ID":"df6a49b3-8428-482c-9472-8a40b5c69f6c","Type":"ContainerStarted","Data":"0962e7fd81c620c9b820d43cd4cba49e58e6a36e1bea56a7c6641cfa794a7995"}
May 20 14:59:01.472136 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:01.472099    2557 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g" event={"ID":"df6a49b3-8428-482c-9472-8a40b5c69f6c","Type":"ContainerStarted","Data":"d498318bde0d933e3b258f0d99b88a6a6ca92f45e31d6b8c8055f48777e6a500"}
May 20 14:59:01.472302 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:01.472177    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:01.492142 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:01.492103    2557 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g" podStartSLOduration=1.492087168 podStartE2EDuration="1.492087168s" podCreationTimestamp="2026-05-20 14:59:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:59:01.490679303 +0000 UTC m=+4121.628233524" watchObservedRunningTime="2026-05-20 14:59:01.492087168 +0000 UTC m=+4121.629641395"
May 20 14:59:01.831402 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:01.831372    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ww2l9_622157d3-3855-4995-8334-d8407eaffade/node-ca/0.log"
May 20 14:59:02.534845 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:02.534814    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75d66f44d8-tqrhq_72fcf5ad-c6b9-47b9-a98e-25767a7ead5a/router/0.log"
May 20 14:59:02.915759 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:02.915730    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4qp54_0cfdbcdd-0be7-42e1-91eb-9892ee0ea925/serve-healthcheck-canary/0.log"
May 20 14:59:03.435181 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:03.435137    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l65x8_4da3b58f-efe6-42c5-baba-7d144e34ebef/kube-rbac-proxy/0.log"
May 20 14:59:03.459713 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:03.459689    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l65x8_4da3b58f-efe6-42c5-baba-7d144e34ebef/exporter/0.log"
May 20 14:59:03.482138 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:03.482107    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l65x8_4da3b58f-efe6-42c5-baba-7d144e34ebef/extractor/0.log"
May 20 14:59:05.937023 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:05.936994    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x6tm9_a7c29e6f-9729-4b30-8ccd-3741d12f722d/s3-init/0.log"
May 20 14:59:05.955362 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:05.955335    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-x6tm9_a7c29e6f-9729-4b30-8ccd-3741d12f722d/download-hf-model/0.log"
May 20 14:59:05.986015 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:05.985988    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-5mc94_9a6aa934-7bc4-4a4c-884e-c0118ed51c5c/s3-tls-init-custom/0.log"
May 20 14:59:06.004417 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:06.004397    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-5mc94_9a6aa934-7bc4-4a4c-884e-c0118ed51c5c/download-hf-model/0.log"
May 20 14:59:06.061511 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:06.061494    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-crtgt_e9c7c931-2114-4774-8579-f89f4efb905f/s3-tls-init-serving/0.log"
May 20 14:59:06.080708 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:06.080688    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-crtgt_e9c7c931-2114-4774-8579-f89f4efb905f/download-hf-model/0.log"
May 20 14:59:07.485494 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:07.485463    2557 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-674pn/perf-node-gather-daemonset-z888g"
May 20 14:59:10.493203 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:10.493177    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-s4nl2_1b095c39-238f-4149-92de-db3dc849e556/migrator/0.log"
May 20 14:59:10.515980 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:10.515959    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-s4nl2_1b095c39-238f-4149-92de-db3dc849e556/graceful-termination/0.log"
May 20 14:59:10.829525 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:10.829488    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-s25dc_42b21860-0b33-45ef-b866-d55adf52bbc8/kube-storage-version-migrator-operator/1.log"
May 20 14:59:10.830479 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:10.830456    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-s25dc_42b21860-0b33-45ef-b866-d55adf52bbc8/kube-storage-version-migrator-operator/0.log"
May 20 14:59:12.137279 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.137252    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-th9gl_c8b51a35-4910-4bcf-9198-c36444ad8dd5/kube-multus-additional-cni-plugins/0.log"
May 20 14:59:12.193871 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.193834    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-th9gl_c8b51a35-4910-4bcf-9198-c36444ad8dd5/egress-router-binary-copy/0.log"
May 20 14:59:12.213086 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.213063    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-th9gl_c8b51a35-4910-4bcf-9198-c36444ad8dd5/cni-plugins/0.log"
May 20 14:59:12.239100 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.239075    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-th9gl_c8b51a35-4910-4bcf-9198-c36444ad8dd5/bond-cni-plugin/0.log"
May 20 14:59:12.266881 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.266842    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-th9gl_c8b51a35-4910-4bcf-9198-c36444ad8dd5/routeoverride-cni/0.log"
May 20 14:59:12.291403 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.291336    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-th9gl_c8b51a35-4910-4bcf-9198-c36444ad8dd5/whereabouts-cni-bincopy/0.log"
May 20 14:59:12.313417 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.313391    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-th9gl_c8b51a35-4910-4bcf-9198-c36444ad8dd5/whereabouts-cni/0.log"
May 20 14:59:12.391076 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.391049    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p9tc9_8be3ff41-5ba1-4283-877d-529de6b51fc7/kube-multus/0.log"
May 20 14:59:12.485921 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.485894    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p9kp9_438841d1-e772-4acf-b0a6-58e5afe1937f/network-metrics-daemon/0.log"
May 20 14:59:12.506211 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:12.506192    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p9kp9_438841d1-e772-4acf-b0a6-58e5afe1937f/kube-rbac-proxy/0.log"
May 20 14:59:13.684233 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:13.684199    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-controller/0.log"
May 20 14:59:13.703558 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:13.703522    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/0.log"
May 20 14:59:13.721486 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:13.721461    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovn-acl-logging/1.log"
May 20 14:59:13.741960 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:13.741940    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/kube-rbac-proxy-node/0.log"
May 20 14:59:13.766497 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:13.766478    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/kube-rbac-proxy-ovn-metrics/0.log"
May 20 14:59:13.790256 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:13.790231    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/northd/0.log"
May 20 14:59:13.814986 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:13.814969    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/nbdb/0.log"
May 20 14:59:13.839951 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:13.839931    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/sbdb/0.log"
May 20 14:59:13.950645 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:13.950568    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gv7zm_3c2fa842-0cdc-445c-ad8b-bc3050ed727a/ovnkube-controller/0.log"
May 20 14:59:15.276392 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:15.276361    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-tz48w_af961ba2-f410-45db-84ef-e8d1c983ed54/check-endpoints/0.log"
May 20 14:59:15.324952 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:15.324926    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wzjlt_36b3619d-2709-46bd-af24-93312d906c20/network-check-target-container/0.log"
May 20 14:59:16.328784 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:16.328760    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-n8fcf_612be1be-2df4-4a7d-a498-87ad24a63762/iptables-alerter/0.log"
May 20 14:59:17.036136 ip-10-0-139-248 kubenswrapper[2557]: I0520 14:59:17.036104    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-62pdb_03b29b31-e26c-4dfc-ba2f-ca176905526b/tuned/0.log"