Apr 24 22:29:56.467650 ip-10-0-134-116 systemd[1]: Starting Kubernetes Kubelet...
Apr 24 22:29:56.852453 ip-10-0-134-116 kubenswrapper[2543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 24 22:29:56.852453 ip-10-0-134-116 kubenswrapper[2543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Apr 24 22:29:56.852453 ip-10-0-134-116 kubenswrapper[2543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 24 22:29:56.852453 ip-10-0-134-116 kubenswrapper[2543]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Apr 24 22:29:56.852453 ip-10-0-134-116 kubenswrapper[2543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 24 22:29:56.854535 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.854451    2543 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Apr 24 22:29:56.857521 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857502    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 22:29:56.857521 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857519    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 22:29:56.857521 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857524    2543 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 22:29:56.857521 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857528    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857534    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857538    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857542    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857546    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857550    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857554    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857558    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857562    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857566    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857571    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857574    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857578    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857583    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857596    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857601    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857604    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857609    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857612    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857617    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 22:29:56.857749 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857620    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857625    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857629    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857634    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857638    2543 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857642    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857647    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857651    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857655    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857659    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857663    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857667    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857671    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857676    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857680    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857685    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857689    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857694    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857698    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857702    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 22:29:56.858327 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857706    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857710    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857714    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857720    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857726    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857731    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857736    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857745    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857749    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857753    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857758    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857762    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857766    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857770    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857775    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857779    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857784    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857788    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857792    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 22:29:56.858871 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857796    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857800    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857805    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857808    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857812    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857817    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857823    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857830    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857834    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857838    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857842    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857847    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857851    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857855    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857859    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857863    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857868    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857872    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857876    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 22:29:56.859333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857880    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857884    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857889    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857893    2543 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.857897    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858368    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858381    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858385    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858388    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858391    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858393    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858397    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858400    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858403    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858405    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858408    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858411    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858413    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858416    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858418    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 22:29:56.859820 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858421    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858423    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858426    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858428    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858430    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858433    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858436    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858438    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858441    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858444    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858446    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858449    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858451    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858454    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858456    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858459    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858461    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858464    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858467    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858470    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 22:29:56.860305 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858472    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858475    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858478    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858480    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858483    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858485    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858488    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858490    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858493    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858496    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858499    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858501    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858504    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858506    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858509    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858511    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858514    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858516    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858519    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 22:29:56.860826 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858521    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858524    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858526    2543 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858529    2543 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858531    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858534    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858536    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858539    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858541    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858544    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858546    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858550    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858553    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858555    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858558    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858561    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858563    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858566    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858568    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858570    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 22:29:56.861287 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858573    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858576    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858579    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858582    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858584    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858587    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858590    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858594    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858597    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858600    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858602    2543 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.858604    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859684    2543 flags.go:64] FLAG: --address="0.0.0.0"
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859695    2543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859703    2543 flags.go:64] FLAG: --anonymous-auth="true"
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859707    2543 flags.go:64] FLAG: --application-metrics-count-limit="100"
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859712    2543 flags.go:64] FLAG: --authentication-token-webhook="false"
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859716    2543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859720    2543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859725    2543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859728    2543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Apr 24 22:29:56.861792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859731    2543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859735    2543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859738    2543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859741    2543 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859744    2543 flags.go:64] FLAG: --cgroup-root=""
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859747    2543 flags.go:64] FLAG: --cgroups-per-qos="true"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859750    2543 flags.go:64] FLAG: --client-ca-file=""
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859753    2543 flags.go:64] FLAG: --cloud-config=""
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859756    2543 flags.go:64] FLAG: --cloud-provider="external"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859758    2543 flags.go:64] FLAG: --cluster-dns="[]"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859763    2543 flags.go:64] FLAG: --cluster-domain=""
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859765    2543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859768    2543 flags.go:64] FLAG: --config-dir=""
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859771    2543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859775    2543 flags.go:64] FLAG: --container-log-max-files="5"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859779    2543 flags.go:64] FLAG: --container-log-max-size="10Mi"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859782    2543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859785    2543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859788    2543 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859791    2543 flags.go:64] FLAG: --contention-profiling="false"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859794    2543 flags.go:64] FLAG: --cpu-cfs-quota="true"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859797    2543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859800    2543 flags.go:64] FLAG: --cpu-manager-policy="none"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859803    2543 flags.go:64] FLAG: --cpu-manager-policy-options=""
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859807    2543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Apr 24 22:29:56.862297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859810    2543 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859813    2543 flags.go:64] FLAG: --enable-debugging-handlers="true"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859816    2543 flags.go:64] FLAG: --enable-load-reader="false"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859820    2543 flags.go:64] FLAG: --enable-server="true"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859823    2543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859827    2543 flags.go:64] FLAG: --event-burst="100"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859831    2543 flags.go:64] FLAG: --event-qps="50"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859834    2543 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859837    2543 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859840    2543 flags.go:64] FLAG: --eviction-hard=""
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859844    2543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859847    2543 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859850    2543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859853    2543 flags.go:64] FLAG: --eviction-soft=""
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859856    2543 flags.go:64] FLAG: --eviction-soft-grace-period=""
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859858    2543 flags.go:64] FLAG: --exit-on-lock-contention="false"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859862    2543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859864    2543 flags.go:64] FLAG: --experimental-mounter-path=""
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859867    2543 flags.go:64] FLAG: --fail-cgroupv1="false"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859870    2543 flags.go:64] FLAG: --fail-swap-on="true"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859873    2543 flags.go:64] FLAG: --feature-gates=""
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859877    2543 flags.go:64] FLAG: --file-check-frequency="20s"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859880    2543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859882    2543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859885    2543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859888    2543 flags.go:64] FLAG: --healthz-port="10248"
Apr 24 22:29:56.862929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859891    2543 flags.go:64] FLAG: --help="false"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859894    2543 flags.go:64] FLAG: --hostname-override="ip-10-0-134-116.ec2.internal"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859897    2543 flags.go:64] FLAG: --housekeeping-interval="10s"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859900    2543 flags.go:64] FLAG: --http-check-frequency="20s"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859903    2543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859906    2543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859909    2543 flags.go:64] FLAG: --image-gc-high-threshold="85"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859912    2543 flags.go:64] FLAG: --image-gc-low-threshold="80"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859915    2543 flags.go:64] FLAG: --image-service-endpoint=""
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859917    2543 flags.go:64] FLAG: --kernel-memcg-notification="false"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859921    2543 flags.go:64] FLAG: --kube-api-burst="100"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859924    2543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859927    2543 flags.go:64] FLAG: --kube-api-qps="50"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859930    2543 flags.go:64] FLAG: --kube-reserved=""
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859933    2543 flags.go:64] FLAG: --kube-reserved-cgroup=""
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859936    2543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859939    2543 flags.go:64] FLAG: --kubelet-cgroups=""
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859942    2543 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859945    2543 flags.go:64] FLAG: --lock-file=""
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859947    2543 flags.go:64] FLAG: --log-cadvisor-usage="false"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859950    2543 flags.go:64] FLAG: --log-flush-frequency="5s"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859953    2543 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859958    2543 flags.go:64] FLAG: --log-json-split-stream="false"
Apr 24 22:29:56.863577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859960    2543 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859963    2543 flags.go:64] FLAG: --log-text-split-stream="false"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859966    2543 flags.go:64] FLAG: --logging-format="text"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859969    2543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859972    2543 flags.go:64] FLAG: --make-iptables-util-chains="true"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859975    2543 flags.go:64] FLAG: --manifest-url=""
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859978    2543 flags.go:64] FLAG: --manifest-url-header=""
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859982    2543 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859985    2543 flags.go:64] FLAG: --max-open-files="1000000"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859992    2543 flags.go:64] FLAG: --max-pods="110"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859995    2543 flags.go:64] FLAG: --maximum-dead-containers="-1"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.859998    2543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860001    2543 flags.go:64] FLAG: --memory-manager-policy="None"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860004    2543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860006    2543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860009    2543 flags.go:64] FLAG: --node-ip="0.0.0.0"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860012    2543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860019    2543 flags.go:64] FLAG: --node-status-max-images="50"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860022    2543 flags.go:64] FLAG: --node-status-update-frequency="10s"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860025    2543 flags.go:64] FLAG: --oom-score-adj="-999"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860028    2543 flags.go:64] FLAG: --pod-cidr=""
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860031    2543 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8cfe89231412ff3ee8cb6207fa0be33cad0f08e88c9c0f1e9f7e8c6f14d6715"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860036    2543 flags.go:64] FLAG: --pod-manifest-path=""
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860039    2543 flags.go:64] FLAG: --pod-max-pids="-1"
Apr 24 22:29:56.864211 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860042    2543 flags.go:64] FLAG: --pods-per-core="0"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860045    2543 flags.go:64] FLAG: --port="10250"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860048    2543 flags.go:64] FLAG: --protect-kernel-defaults="false"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860051    2543 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ac0f4c84c4b6f16e"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860054    2543 flags.go:64] FLAG: --qos-reserved=""
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860057    2543 flags.go:64] FLAG: --read-only-port="10255"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860059    2543 flags.go:64] FLAG: --register-node="true"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860062    2543 flags.go:64] FLAG: --register-schedulable="true"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860065    2543 flags.go:64] FLAG: --register-with-taints=""
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860069    2543 flags.go:64] FLAG: --registry-burst="10"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860071    2543 flags.go:64] FLAG: --registry-qps="5"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860074    2543 flags.go:64] FLAG: --reserved-cpus=""
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860077    2543 flags.go:64] FLAG: --reserved-memory=""
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860080    2543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860083    2543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860086    2543 flags.go:64] FLAG: --rotate-certificates="false"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860089    2543 flags.go:64] FLAG: --rotate-server-certificates="false"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860092    2543 flags.go:64] FLAG: --runonce="false"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860095    2543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860098    2543 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860101    2543 flags.go:64] FLAG: --seccomp-default="false"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860104    2543 flags.go:64] FLAG: --serialize-image-pulls="true"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860107    2543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860110    2543 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860113    2543 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860116    2543 flags.go:64] FLAG: --storage-driver-password="root"
Apr 24 22:29:56.864848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860119    2543 flags.go:64] FLAG: --storage-driver-secure="false"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860122    2543 flags.go:64] FLAG: --storage-driver-table="stats"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860124    2543 flags.go:64] FLAG: --storage-driver-user="root"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860127    2543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860131    2543 flags.go:64] FLAG: --sync-frequency="1m0s"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860133    2543 flags.go:64] FLAG: --system-cgroups=""
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860136    2543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860141    2543 flags.go:64] FLAG: --system-reserved-cgroup=""
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860144    2543 flags.go:64] FLAG: --tls-cert-file=""
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860147    2543 flags.go:64] FLAG: --tls-cipher-suites="[]"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860151    2543 flags.go:64] FLAG: --tls-min-version=""
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860154    2543 flags.go:64] FLAG: --tls-private-key-file=""
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860157    2543 flags.go:64] FLAG: --topology-manager-policy="none"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860159    2543 flags.go:64] FLAG: --topology-manager-policy-options=""
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860162    2543 flags.go:64] FLAG: --topology-manager-scope="container"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860165    2543 flags.go:64] FLAG: --v="2"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860169    2543 flags.go:64] FLAG: --version="false"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860176    2543 flags.go:64] FLAG: --vmodule=""
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860180    2543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.860183    2543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860272    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860277    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860280    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860282    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 22:29:56.865499 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860285    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860288    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860291    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860293    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860296    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860298    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860300    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860303    2543 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860306    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860309    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860311    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860314    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860317    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860320    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860323    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860325    2543 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860328    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860330    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860333    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860352    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860355    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 22:29:56.866122 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860358    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860361    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860363    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860366    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860370    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860373    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860375    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860378    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860380    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860383    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860385    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860388    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860390    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860394    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860397    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860399    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860402    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860404    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860407    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 22:29:56.866683 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860410    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860412    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860415    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860417    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860422    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860426    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860429    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860432    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860435    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860438    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860440    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860442    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860445    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860447    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860450    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860452    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860455    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860459    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860461    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860464    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 22:29:56.867295 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860466    2543 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860468    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860471    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860473    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860476    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860478    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860480    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860483    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860485    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860487    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860490    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860493    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860495    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860498    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860500    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860503    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860505    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860508    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860510    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 22:29:56.867857 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860513    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 22:29:56.868332 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860515    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 22:29:56.868332 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.860518    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 22:29:56.868332 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.861011    2543 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 24 22:29:56.870780 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.870758    2543 server.go:530] "Kubelet version" kubeletVersion="v1.33.9"
Apr 24 22:29:56.870780 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.870778    2543 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870824    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870830    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870833    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870836    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870839    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870842    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870845    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870847    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870850    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870853    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870856    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870861    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870863    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870866    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870869    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870871    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870874    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870877    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 22:29:56.870882 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870881    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870883    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870886    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870889    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870892    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870895    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870897    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870900    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870902    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870905    2543 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870907    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870910    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870913    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870915    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870918    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870920    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870923    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870925    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870928    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870930    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 22:29:56.871381 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870933    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870935    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870938    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870940    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870943    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870946    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870948    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870951    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870953    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870956    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870958    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870961    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870963    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870966    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870969    2543 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870972    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870974    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870978    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870982    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 22:29:56.871908 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870985    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870988    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870990    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870993    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870995    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.870998    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871000    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871002    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871005    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871007    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871010    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871013    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871015    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871018    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871020    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871022    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871025    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871027    2543 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871030    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871033    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 22:29:56.872398 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871035    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871038    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871040    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871043    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871045    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871048    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871050    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871054    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871056    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.871061    2543 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871152    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871156    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871158    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871161    2543 feature_gate.go:328] unrecognized feature gate: Example2
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871164    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871166    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 24 22:29:56.872894 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871169    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871172    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871174    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871177    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871179    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871181    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871184    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871187    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871189    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871192    2543 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871194    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871197    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871200    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871202    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871205    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871207    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871210    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871212    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871215    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871217    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 24 22:29:56.873293 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871220    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871222    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871225    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871228    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871232    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871235    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871238    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871241    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871244    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871247    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871250    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871254    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871258    2543 feature_gate.go:328] unrecognized feature gate: Example
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871262    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871265    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871269    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871273    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871276    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871279    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871281    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 24 22:29:56.873794 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871284    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871286    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871289    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871292    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871294    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871297    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871299    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871302    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871304    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871308    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871310    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871313    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871315    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871318    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871320    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871324    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871327    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871329    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871331    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 24 22:29:56.874262 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871354    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871357    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871360    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871363    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871365    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871368    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871370    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871373    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871376    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871378    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871381    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871383    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871386    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871388    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871392    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871396    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871398    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871411    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871413    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871416    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 24 22:29:56.874740 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:56.871418    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 24 22:29:56.875214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.871424    2543 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 24 22:29:56.875214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.872041    2543 server.go:962] "Client rotation is on, will bootstrap in background"
Apr 24 22:29:56.875214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.873879    2543 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Apr 24 22:29:56.875214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.874690    2543 server.go:1019] "Starting client certificate rotation"
Apr 24 22:29:56.875214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.874786    2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 24 22:29:56.875214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.874826    2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 24 22:29:56.897322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.897309    2543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 24 22:29:56.899617 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.899602    2543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 24 22:29:56.911464 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.911443    2543 log.go:25] "Validated CRI v1 runtime API"
Apr 24 22:29:56.916783 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.916771    2543 log.go:25] "Validated CRI v1 image API"
Apr 24 22:29:56.918121 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.918108    2543 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Apr 24 22:29:56.926278 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.926260    2543 fs.go:135] Filesystem UUIDs: map[7016b227-f2f5-4173-8e42-aecd14c439e3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e8b3409f-e0e7-46fc-8d43-501567e78f98:/dev/nvme0n1p3]
Apr 24 22:29:56.926357 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.926279    2543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Apr 24 22:29:56.928947 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.928922    2543 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 24 22:29:56.932638 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.932528    2543 manager.go:217] Machine: {Timestamp:2026-04-24 22:29:56.930781191 +0000 UTC m=+0.357178712 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100409 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ae8a13498746e0d673f5b02b60bff SystemUUID:ec2ae8a1-3498-746e-0d67-3f5b02b60bff BootID:e25c0958-4eae-4766-9c18-1f18a19e24e6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:85:00:63:24:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:85:00:63:24:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:34:58:76:54:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 24 22:29:56.932638 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.932628    2543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Apr 24 22:29:56.932770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.932702    2543 manager.go:233] Version: {KernelVersion:5.14.0-570.107.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260414-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Apr 24 22:29:56.933032 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.933014    2543 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Apr 24 22:29:56.933175 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.933035    2543 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-116.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Apr 24 22:29:56.933218 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.933184    2543 topology_manager.go:138] "Creating topology manager with none policy"
Apr 24 22:29:56.933218 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.933193    2543 container_manager_linux.go:306] "Creating device plugin manager"
Apr 24 22:29:56.933218 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.933206    2543 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 24 22:29:56.934002 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.933991    2543 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 24 22:29:56.935186 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.935176    2543 state_mem.go:36] "Initialized new in-memory state store"
Apr 24 22:29:56.935293 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.935284    2543 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Apr 24 22:29:56.937634 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.937624    2543 kubelet.go:491] "Attempting to sync node with API server"
Apr 24 22:29:56.937672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.937644    2543 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Apr 24 22:29:56.937672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.937655    2543 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Apr 24 22:29:56.937672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.937664    2543 kubelet.go:397] "Adding apiserver pod source"
Apr 24 22:29:56.937672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.937671    2543 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Apr 24 22:29:56.938652 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.938638    2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 24 22:29:56.938696 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.938664    2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 24 22:29:56.941669 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.941654    2543 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1"
Apr 24 22:29:56.942839 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.942826    2543 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Apr 24 22:29:56.944438 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944426    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Apr 24 22:29:56.944481 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944444    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Apr 24 22:29:56.944481 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944451    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Apr 24 22:29:56.944481 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944458    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Apr 24 22:29:56.944481 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944466    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Apr 24 22:29:56.944481 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944475    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Apr 24 22:29:56.944481 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944480    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Apr 24 22:29:56.944639 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944486    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Apr 24 22:29:56.944639 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944494    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Apr 24 22:29:56.944639 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944499    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Apr 24 22:29:56.944639 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944508    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Apr 24 22:29:56.944639 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.944517    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Apr 24 22:29:56.945765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.945756    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Apr 24 22:29:56.945803 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.945768    2543 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Apr 24 22:29:56.948255 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.948239    2543 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-116.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Apr 24 22:29:56.948515 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:56.948490    2543 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Apr 24 22:29:56.948775 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:56.948754    2543 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Apr 24 22:29:56.949607 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.949594    2543 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Apr 24 22:29:56.949668 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.949629    2543 server.go:1295] "Started kubelet"
Apr 24 22:29:56.949711 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.949695    2543 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Apr 24 22:29:56.949773 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.949725    2543 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Apr 24 22:29:56.949806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.949799    2543 server_v1.go:47] "podresources" method="list" useActivePods=true
Apr 24 22:29:56.950408 ip-10-0-134-116 systemd[1]: Started Kubernetes Kubelet.
Apr 24 22:29:56.951212 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.951153    2543 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Apr 24 22:29:56.952498 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.952484    2543 server.go:317] "Adding debug handlers to kubelet server"
Apr 24 22:29:56.956372 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.956350    2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Apr 24 22:29:56.956824 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.956807    2543 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Apr 24 22:29:56.957775 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.957704    2543 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Apr 24 22:29:56.958147 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:56.958026    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:56.958821 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.958804    2543 volume_manager.go:295] "The desired_state_of_world populator starts"
Apr 24 22:29:56.958902 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.958826    2543 volume_manager.go:297] "Starting Kubelet Volume Manager"
Apr 24 22:29:56.958971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.958953    2543 reconstruct.go:97] "Volume reconstruction finished"
Apr 24 22:29:56.958971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.958962    2543 reconciler.go:26] "Reconciler: start to sync state"
Apr 24 22:29:56.959139 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.959122    2543 factory.go:55] Registering systemd factory
Apr 24 22:29:56.959213 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.959144    2543 factory.go:223] Registration of the systemd container factory successfully
Apr 24 22:29:56.959483 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:56.959463    2543 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Apr 24 22:29:56.960096 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.960078    2543 factory.go:153] Registering CRI-O factory
Apr 24 22:29:56.960096 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.960099    2543 factory.go:223] Registration of the crio container factory successfully
Apr 24 22:29:56.960223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.960160    2543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Apr 24 22:29:56.960223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.960183    2543 factory.go:103] Registering Raw factory
Apr 24 22:29:56.960223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.960197    2543 manager.go:1196] Started watching for new ooms in manager
Apr 24 22:29:56.960512 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:56.959608    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-116.ec2.internal.18a96b90a441c057  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-116.ec2.internal,UID:ip-10-0-134-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-116.ec2.internal,},FirstTimestamp:2026-04-24 22:29:56.949606487 +0000 UTC m=+0.376004007,LastTimestamp:2026-04-24 22:29:56.949606487 +0000 UTC m=+0.376004007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-116.ec2.internal,}"
Apr 24 22:29:56.960570 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.960549    2543 manager.go:319] Starting recovery of all containers
Apr 24 22:29:56.961184 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:56.960078    2543 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Apr 24 22:29:56.961943 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:56.961909    2543 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Apr 24 22:29:56.969673 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.969517    2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pp8wb"
Apr 24 22:29:56.973322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.973308    2543 manager.go:324] Recovery completed
Apr 24 22:29:56.977399 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.977387    2543 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 22:29:56.979602 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.979575    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 22:29:56.979689 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.979607    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 22:29:56.979689 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.979621    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Apr 24 22:29:56.980097 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.980080    2543 cpu_manager.go:222] "Starting CPU manager" policy="none"
Apr 24 22:29:56.980097 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.980094    2543 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Apr 24 22:29:56.980195 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.980110    2543 state_mem.go:36] "Initialized new in-memory state store"
Apr 24 22:29:56.981055 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.981039    2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pp8wb"
Apr 24 22:29:56.981778 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:56.981721    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-116.ec2.internal.18a96b90a60b4839  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-116.ec2.internal,UID:ip-10-0-134-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-116.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-116.ec2.internal,},FirstTimestamp:2026-04-24 22:29:56.979591225 +0000 UTC m=+0.405988746,LastTimestamp:2026-04-24 22:29:56.979591225 +0000 UTC m=+0.405988746,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-116.ec2.internal,}"
Apr 24 22:29:56.982659 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.982646    2543 policy_none.go:49] "None policy: Start"
Apr 24 22:29:56.982713 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.982662    2543 memory_manager.go:186] "Starting memorymanager" policy="None"
Apr 24 22:29:56.982713 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:56.982674    2543 state_mem.go:35] "Initializing new in-memory state store"
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.024861    2543 manager.go:341] "Starting Device Plugin manager"
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.024891    2543 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.024901    2543 server.go:85] "Starting device plugin registration server"
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.025106    2543 eviction_manager.go:189] "Eviction manager: starting control loop"
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.025118    2543 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.025188    2543 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.025267    2543 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.025276    2543 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.025727    2543 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Apr 24 22:29:57.036798 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.025765    2543 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.049333 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.049309    2543 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Apr 24 22:29:57.050575 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.050558    2543 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Apr 24 22:29:57.050654 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.050578    2543 status_manager.go:230] "Starting to sync pod status with apiserver"
Apr 24 22:29:57.050654 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.050592    2543 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Apr 24 22:29:57.050654 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.050598    2543 kubelet.go:2451] "Starting kubelet main sync loop"
Apr 24 22:29:57.050654 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.050630    2543 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Apr 24 22:29:57.052928 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.052910    2543 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 22:29:57.125243 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.125197    2543 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 22:29:57.126085 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.126071    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 22:29:57.126154 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.126099    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 22:29:57.126154 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.126119    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Apr 24 22:29:57.126154 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.126140    2543 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.132167 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.132147    2543 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.132167 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.132167    2543 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-116.ec2.internal\": node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.151170 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.151151    2543 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"]
Apr 24 22:29:57.151248 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.151215    2543 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 22:29:57.151849 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.151836    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 22:29:57.151898 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.151861    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 22:29:57.151898 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.151875    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Apr 24 22:29:57.154003 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.153989    2543 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 22:29:57.154183 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.154168    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.154250 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.154205    2543 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 22:29:57.154677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.154653    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 22:29:57.154677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.154679    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 22:29:57.154793 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.154689    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Apr 24 22:29:57.154793 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.154660    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 22:29:57.154793 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.154762    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 22:29:57.154793 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.154776    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Apr 24 22:29:57.156998 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.156983    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.157062 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.157019    2543 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 24 22:29:57.157683 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.157662    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Apr 24 22:29:57.157765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.157689    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Apr 24 22:29:57.157765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.157699    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Apr 24 22:29:57.159327 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.159307    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de4cd73d2cc50d777b3fcaab17c0aa50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"de4cd73d2cc50d777b3fcaab17c0aa50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.159413 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.159364    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de4cd73d2cc50d777b3fcaab17c0aa50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"de4cd73d2cc50d777b3fcaab17c0aa50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.159460 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.159423    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60d03c02f9ed5519c8c0179698b80bb4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-116.ec2.internal\" (UID: \"60d03c02f9ed5519c8c0179698b80bb4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.162544 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.162527    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.179954 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.179939    2543 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-116.ec2.internal\" not found" node="ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.184102 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.184085    2543 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-116.ec2.internal\" not found" node="ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.259776 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.259753    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de4cd73d2cc50d777b3fcaab17c0aa50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"de4cd73d2cc50d777b3fcaab17c0aa50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.259865 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.259783    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de4cd73d2cc50d777b3fcaab17c0aa50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"de4cd73d2cc50d777b3fcaab17c0aa50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.259865 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.259808    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60d03c02f9ed5519c8c0179698b80bb4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-116.ec2.internal\" (UID: \"60d03c02f9ed5519c8c0179698b80bb4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.259865 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.259845    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de4cd73d2cc50d777b3fcaab17c0aa50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"de4cd73d2cc50d777b3fcaab17c0aa50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.259865 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.259852    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de4cd73d2cc50d777b3fcaab17c0aa50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"de4cd73d2cc50d777b3fcaab17c0aa50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.260009 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.259878    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60d03c02f9ed5519c8c0179698b80bb4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-116.ec2.internal\" (UID: \"60d03c02f9ed5519c8c0179698b80bb4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.262867 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.262853    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.363512 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.363491    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.464210 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.464167    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.482353 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.482320    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.486673 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.486658    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:57.564454 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.564431    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.664988 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.664963    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.766041 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.765991    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.766731 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.766717    2543 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 22:29:57.866530 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.866506    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.874668 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.874647    2543 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Apr 24 22:29:57.874794 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.874776    2543 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 24 22:29:57.874834 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.874778    2543 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 24 22:29:57.957196 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.957172    2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Apr 24 22:29:57.966795 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:57.966769    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:57.971625 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.971604    2543 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 24 22:29:57.982711 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.982683    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-23 22:24:56 +0000 UTC" deadline="2027-10-14 18:25:27.144212181 +0000 UTC"
Apr 24 22:29:57.982711 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.982707    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12907h55m29.161507829s"
Apr 24 22:29:57.995929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:57.995904    2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fhrgb"
Apr 24 22:29:58.006628 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.006604    2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fhrgb"
Apr 24 22:29:58.037629 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:58.037604    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d03c02f9ed5519c8c0179698b80bb4.slice/crio-6a813213aefcbff5c344b420ead2646f28b440d70640b44c52617d24b8c8930e WatchSource:0}: Error finding container 6a813213aefcbff5c344b420ead2646f28b440d70640b44c52617d24b8c8930e: Status 404 returned error can't find the container with id 6a813213aefcbff5c344b420ead2646f28b440d70640b44c52617d24b8c8930e
Apr 24 22:29:58.037920 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:58.037901    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4cd73d2cc50d777b3fcaab17c0aa50.slice/crio-52f911abba0f9bdd1d17fe7a2f39018351322d01450b236f4deb7da483ce0777 WatchSource:0}: Error finding container 52f911abba0f9bdd1d17fe7a2f39018351322d01450b236f4deb7da483ce0777: Status 404 returned error can't find the container with id 52f911abba0f9bdd1d17fe7a2f39018351322d01450b236f4deb7da483ce0777
Apr 24 22:29:58.042094 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.042080    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 22:29:58.053148 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.053101    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal" event={"ID":"60d03c02f9ed5519c8c0179698b80bb4","Type":"ContainerStarted","Data":"6a813213aefcbff5c344b420ead2646f28b440d70640b44c52617d24b8c8930e"}
Apr 24 22:29:58.054196 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.054174    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal" event={"ID":"de4cd73d2cc50d777b3fcaab17c0aa50","Type":"ContainerStarted","Data":"52f911abba0f9bdd1d17fe7a2f39018351322d01450b236f4deb7da483ce0777"}
Apr 24 22:29:58.067701 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:58.067681    2543 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Apr 24 22:29:58.142020 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.142000    2543 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 22:29:58.157956 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.157939    2543 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:58.171466 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.171446    2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 24 22:29:58.172263 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.172252    2543 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Apr 24 22:29:58.185222 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.185206    2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 24 22:29:58.507306 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.507240    2543 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 22:29:58.939043 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.938969    2543 apiserver.go:52] "Watching apiserver"
Apr 24 22:29:58.949444 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.949423    2543 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Apr 24 22:29:58.951116 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.951095    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pxq7f","openshift-network-diagnostics/network-check-target-2lldp","kube-system/konnectivity-agent-zkmd6","openshift-image-registry/node-ca-2kt8q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal","openshift-network-operator/iptables-alerter-9gkvt","openshift-ovn-kubernetes/ovnkube-node-t72f2","kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s","openshift-cluster-node-tuning-operator/tuned-p8kkf","openshift-dns/node-resolver-8tz2q","openshift-multus/multus-978tk","openshift-multus/multus-additional-cni-plugins-k2kcl"]
Apr 24 22:29:58.953680 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.953659    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.956600 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.956579    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:29:58.956702 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:58.956658    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:29:58.957971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.957951    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Apr 24 22:29:58.958095 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.957958    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Apr 24 22:29:58.958648 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.958628    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Apr 24 22:29:58.958716 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.958660    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Apr 24 22:29:58.958857 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.958839    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Apr 24 22:29:58.959412 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.959378    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Apr 24 22:29:58.959412 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.959405    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ztw56\""
Apr 24 22:29:58.960927 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.960910    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:58.964205 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.963550    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:29:58.964205 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.963644    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:58.965850 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.965832    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:29:58.965940 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:58.965899    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:29:58.966561 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.966539    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Apr 24 22:29:58.966642 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.966625    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wwb2r\""
Apr 24 22:29:58.966737 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.966546    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Apr 24 22:29:58.966831 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.966816    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Apr 24 22:29:58.967624 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967605    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-kubelet\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.967703 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967648    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-slash\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.967703 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967674    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-cni-netd\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.967806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967703    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:29:58.967806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967732    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e170e736-789d-406e-a238-bcaa098a2309-agent-certs\") pod \"konnectivity-agent-zkmd6\" (UID: \"e170e736-789d-406e-a238-bcaa098a2309\") " pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:29:58.967806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967757    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-run-ovn\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.967806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967782    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-node-log\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.967974 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967808    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b7de9be-e366-4f28-8f32-0587de85e41b-env-overrides\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.967974 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967837    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b7de9be-e366-4f28-8f32-0587de85e41b-ovnkube-script-lib\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.967974 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.967857    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e170e736-789d-406e-a238-bcaa098a2309-konnectivity-ca\") pod \"konnectivity-agent-zkmd6\" (UID: \"e170e736-789d-406e-a238-bcaa098a2309\") " pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:29:58.968136 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.968119    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:58.969222 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969197    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-systemd-units\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969292 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969246    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-run-systemd\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969292 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969274    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-var-lib-openvswitch\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969456 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969304    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-run-openvswitch\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969456 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969330    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-log-socket\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969456 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969406    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-cni-bin\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969458    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969486    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-run-netns\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969502    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-etc-openvswitch\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969518    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-run-ovn-kubernetes\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969534    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b7de9be-e366-4f28-8f32-0587de85e41b-ovnkube-config\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969551    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b7de9be-e366-4f28-8f32-0587de85e41b-ovn-node-metrics-cert\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.969723 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.969565    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2lmn\" (UniqueName: \"kubernetes.io/projected/8b7de9be-e366-4f28-8f32-0587de85e41b-kube-api-access-t2lmn\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:58.970299 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.970277    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Apr 24 22:29:58.970299 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.970297    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Apr 24 22:29:58.970652 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.970637    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:58.971534 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.971511    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-crzrv\""
Apr 24 22:29:58.972677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.972551    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Apr 24 22:29:58.972957 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.972893    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Apr 24 22:29:58.974364 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.973659    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5r9c4\""
Apr 24 22:29:58.974364 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.973798    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:58.974364 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.973826    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Apr 24 22:29:58.974364 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.973890    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9nvt5\""
Apr 24 22:29:58.974364 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.974095    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Apr 24 22:29:58.974364 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.974111    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Apr 24 22:29:58.974364 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.974142    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Apr 24 22:29:58.974746 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.974472    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Apr 24 22:29:58.974746 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.974578    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gvh2n\""
Apr 24 22:29:58.974746 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.974612    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Apr 24 22:29:58.976591 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.976571    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-978tk"
Apr 24 22:29:58.977813 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.977793    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Apr 24 22:29:58.978996 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.978976    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:58.983908 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.983885    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Apr 24 22:29:58.984398 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.984379    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Apr 24 22:29:58.984498 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.984473    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Apr 24 22:29:58.984565 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.984383    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Apr 24 22:29:58.984640 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.984624    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Apr 24 22:29:58.984736 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.984713    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f49vr\""
Apr 24 22:29:58.985040 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.984895    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8747p\""
Apr 24 22:29:58.985040 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.984903    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jbwlx\""
Apr 24 22:29:58.985040 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.984954    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Apr 24 22:29:58.986101 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:58.986081    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Apr 24 22:29:59.007318 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.007293    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-23 22:24:57 +0000 UTC" deadline="2028-01-09 02:33:32.264409724 +0000 UTC"
Apr 24 22:29:59.007318 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.007318    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14980h3m33.25709598s"
Apr 24 22:29:59.059331 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.059307    2543 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Apr 24 22:29:59.070042 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070007    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-systemd-units\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070056    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-run-systemd\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070065    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-systemd-units\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070085    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-run-openvswitch\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070118    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-run-systemd\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070128    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-run-openvswitch\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070155    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-device-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070182    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-systemd\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070197    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-tuned\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070216    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-run-ovn-kubernetes\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070231    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b7de9be-e366-4f28-8f32-0587de85e41b-ovnkube-config\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070262    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b7de9be-e366-4f28-8f32-0587de85e41b-ovn-node-metrics-cert\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070270    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-run-ovn-kubernetes\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070279    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070294    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-sys-fs\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070309    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070362    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-sysconfig\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.070384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070384    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkvt\" (UniqueName: \"kubernetes.io/projected/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-kube-api-access-cvkvt\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070403    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-kubelet\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070427    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-sysctl-conf\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070456    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-kubelet\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070472    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-node-log\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070508    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b7de9be-e366-4f28-8f32-0587de85e41b-env-overrides\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070526    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxj9\" (UniqueName: \"kubernetes.io/projected/db346dd5-9969-4a80-9cd0-5c8847bf7d6e-kube-api-access-4qxj9\") pod \"iptables-alerter-9gkvt\" (UID: \"db346dd5-9969-4a80-9cd0-5c8847bf7d6e\") " pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070541    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-node-log\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070545    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-registration-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070614    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-cnibin\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070643    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/676d2ff5-547e-4f1e-a1ca-b87580102275-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070673    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4369d13e-0dde-416a-b48e-2fec6e7a3317-cni-binary-copy\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070703    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-log-socket\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070727    2543 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Apr 24 22:29:59.070748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070737    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070784    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070807    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/676d2ff5-547e-4f1e-a1ca-b87580102275-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070833    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8b7de9be-e366-4f28-8f32-0587de85e41b-ovnkube-config\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070827    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-log-socket\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070856    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7j65\" (UniqueName: \"kubernetes.io/projected/bb485ff8-22b6-40e7-ab56-e64e94f03cbe-kube-api-access-s7j65\") pod \"node-ca-2kt8q\" (UID: \"bb485ff8-22b6-40e7-ab56-e64e94f03cbe\") " pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070869    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8b7de9be-e366-4f28-8f32-0587de85e41b-env-overrides\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070887    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-sys\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070914    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-system-cni-dir\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070945    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.070986    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n8mq\" (UniqueName: \"kubernetes.io/projected/99e7bf73-8fe8-429f-b001-ed59f4e2bd01-kube-api-access-9n8mq\") pod \"node-resolver-8tz2q\" (UID: \"99e7bf73-8fe8-429f-b001-ed59f4e2bd01\") " pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071029    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzq5\" (UniqueName: \"kubernetes.io/projected/676d2ff5-547e-4f1e-a1ca-b87580102275-kube-api-access-hnzq5\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071069    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb485ff8-22b6-40e7-ab56-e64e94f03cbe-host\") pod \"node-ca-2kt8q\" (UID: \"bb485ff8-22b6-40e7-ab56-e64e94f03cbe\") " pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071087    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-tmp\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071104    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db346dd5-9969-4a80-9cd0-5c8847bf7d6e-host-slash\") pod \"iptables-alerter-9gkvt\" (UID: \"db346dd5-9969-4a80-9cd0-5c8847bf7d6e\") " pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071124    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-run\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071146    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-run-netns\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.071388 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071165    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zskqc\" (UniqueName: \"kubernetes.io/projected/4369d13e-0dde-416a-b48e-2fec6e7a3317-kube-api-access-zskqc\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071186    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-run-ovn\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071205    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b7de9be-e366-4f28-8f32-0587de85e41b-ovnkube-script-lib\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071227    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e170e736-789d-406e-a238-bcaa098a2309-konnectivity-ca\") pod \"konnectivity-agent-zkmd6\" (UID: \"e170e736-789d-406e-a238-bcaa098a2309\") " pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071243    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-run-ovn\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071248    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db346dd5-9969-4a80-9cd0-5c8847bf7d6e-iptables-alerter-script\") pod \"iptables-alerter-9gkvt\" (UID: \"db346dd5-9969-4a80-9cd0-5c8847bf7d6e\") " pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071290    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-kubernetes\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071315    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-cnibin\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071357    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-var-lib-cni-multus\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071401    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t2lmn\" (UniqueName: \"kubernetes.io/projected/8b7de9be-e366-4f28-8f32-0587de85e41b-kube-api-access-t2lmn\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071433    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/676d2ff5-547e-4f1e-a1ca-b87580102275-cni-binary-copy\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071454    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb485ff8-22b6-40e7-ab56-e64e94f03cbe-serviceca\") pod \"node-ca-2kt8q\" (UID: \"bb485ff8-22b6-40e7-ab56-e64e94f03cbe\") " pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071475    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-var-lib-kubelet\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071498    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-var-lib-kubelet\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071553    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-hostroot\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071609    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-daemon-config\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071652    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-lib-modules\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.072106 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071702    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-cni-dir\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071718    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8b7de9be-e366-4f28-8f32-0587de85e41b-ovnkube-script-lib\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071767    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e170e736-789d-406e-a238-bcaa098a2309-konnectivity-ca\") pod \"konnectivity-agent-zkmd6\" (UID: \"e170e736-789d-406e-a238-bcaa098a2309\") " pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071829    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-slash\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071837    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-slash\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071865    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e170e736-789d-406e-a238-bcaa098a2309-agent-certs\") pod \"konnectivity-agent-zkmd6\" (UID: \"e170e736-789d-406e-a238-bcaa098a2309\") " pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071889    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-socket-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071909    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-system-cni-dir\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071930    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-var-lib-cni-bin\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071952    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-var-lib-openvswitch\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.071974    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-cni-bin\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072015    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072047    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-cni-bin\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072038    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072064    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-var-lib-openvswitch\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072076    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-host\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072099    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-run-multus-certs\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.072687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072123    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-etc-kubernetes\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072158    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-run-netns\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072187    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-etc-openvswitch\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072195    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-run-netns\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072213    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e7bf73-8fe8-429f-b001-ed59f4e2bd01-tmp-dir\") pod \"node-resolver-8tz2q\" (UID: \"99e7bf73-8fe8-429f-b001-ed59f4e2bd01\") " pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072228    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-etc-openvswitch\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072237    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-os-release\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072271    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-socket-dir-parent\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072302    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-cni-netd\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072327    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tjgg\" (UniqueName: \"kubernetes.io/projected/146efc86-191d-4226-adfb-f36ee7ad4e1a-kube-api-access-4tjgg\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072376    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8b7de9be-e366-4f28-8f32-0587de85e41b-host-cni-netd\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072380    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99e7bf73-8fe8-429f-b001-ed59f4e2bd01-hosts-file\") pod \"node-resolver-8tz2q\" (UID: \"99e7bf73-8fe8-429f-b001-ed59f4e2bd01\") " pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072404    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-sysctl-d\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072448    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-conf-dir\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072471    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8vc\" (UniqueName: \"kubernetes.io/projected/df793786-616a-4001-9c32-2e4ebd101bb2-kube-api-access-8f8vc\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072492    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-modprobe-d\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072512    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-os-release\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.073300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.072534    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-run-k8s-cni-cncf-io\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.074331 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.074308    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b7de9be-e366-4f28-8f32-0587de85e41b-ovn-node-metrics-cert\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.074456 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.074427    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e170e736-789d-406e-a238-bcaa098a2309-agent-certs\") pod \"konnectivity-agent-zkmd6\" (UID: \"e170e736-789d-406e-a238-bcaa098a2309\") " pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:29:59.095638 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.095614    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 22:29:59.095638 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.095639    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 22:29:59.095804 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.095652    2543 projected.go:194] Error preparing data for projected volume kube-api-access-78bxk for pod openshift-network-diagnostics/network-check-target-2lldp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:29:59.095804 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.095727    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk podName:e7d5adb7-ab2b-47df-859a-74f9de07e09b nodeName:}" failed. No retries permitted until 2026-04-24 22:29:59.595696764 +0000 UTC m=+3.022094281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-78bxk" (UniqueName: "kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk") pod "network-check-target-2lldp" (UID: "e7d5adb7-ab2b-47df-859a-74f9de07e09b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:29:59.097655 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.097636    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2lmn\" (UniqueName: \"kubernetes.io/projected/8b7de9be-e366-4f28-8f32-0587de85e41b-kube-api-access-t2lmn\") pod \"ovnkube-node-t72f2\" (UID: \"8b7de9be-e366-4f28-8f32-0587de85e41b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.117143 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.117126    2543 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 22:29:59.173218 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173194    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-tuned\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.173329 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173225    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:29:59.173329 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173251    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-sys-fs\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.173329 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173274    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.173329 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173298    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-sysconfig\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.173329 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173323    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkvt\" (UniqueName: \"kubernetes.io/projected/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-kube-api-access-cvkvt\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173363    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-sysctl-conf\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.173372    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173379    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-sys-fs\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173391    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxj9\" (UniqueName: \"kubernetes.io/projected/db346dd5-9969-4a80-9cd0-5c8847bf7d6e-kube-api-access-4qxj9\") pod \"iptables-alerter-9gkvt\" (UID: \"db346dd5-9969-4a80-9cd0-5c8847bf7d6e\") " pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.173442    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs podName:df793786-616a-4001-9c32-2e4ebd101bb2 nodeName:}" failed. No retries permitted until 2026-04-24 22:29:59.673422066 +0000 UTC m=+3.099819591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs") pod "network-metrics-daemon-pxq7f" (UID: "df793786-616a-4001-9c32-2e4ebd101bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173466    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-registration-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173482    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173497    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-cnibin\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173528    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/676d2ff5-547e-4f1e-a1ca-b87580102275-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173557    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4369d13e-0dde-416a-b48e-2fec6e7a3317-cni-binary-copy\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173583    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/676d2ff5-547e-4f1e-a1ca-b87580102275-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.173611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173608    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s7j65\" (UniqueName: \"kubernetes.io/projected/bb485ff8-22b6-40e7-ab56-e64e94f03cbe-kube-api-access-s7j65\") pod \"node-ca-2kt8q\" (UID: \"bb485ff8-22b6-40e7-ab56-e64e94f03cbe\") " pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173622    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-sysconfig\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173630    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-sys\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173656    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-system-cni-dir\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173681    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-registration-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173692    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8mq\" (UniqueName: \"kubernetes.io/projected/99e7bf73-8fe8-429f-b001-ed59f4e2bd01-kube-api-access-9n8mq\") pod \"node-resolver-8tz2q\" (UID: \"99e7bf73-8fe8-429f-b001-ed59f4e2bd01\") " pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173718    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzq5\" (UniqueName: \"kubernetes.io/projected/676d2ff5-547e-4f1e-a1ca-b87580102275-kube-api-access-hnzq5\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173717    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-sysctl-conf\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173722    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-cnibin\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173745    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb485ff8-22b6-40e7-ab56-e64e94f03cbe-host\") pod \"node-ca-2kt8q\" (UID: \"bb485ff8-22b6-40e7-ab56-e64e94f03cbe\") " pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173786    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-system-cni-dir\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173790    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-sys\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173824    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb485ff8-22b6-40e7-ab56-e64e94f03cbe-host\") pod \"node-ca-2kt8q\" (UID: \"bb485ff8-22b6-40e7-ab56-e64e94f03cbe\") " pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.173950    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-tmp\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174020    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db346dd5-9969-4a80-9cd0-5c8847bf7d6e-host-slash\") pod \"iptables-alerter-9gkvt\" (UID: \"db346dd5-9969-4a80-9cd0-5c8847bf7d6e\") " pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174042    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-run\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174066    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-run-netns\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174092    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zskqc\" (UniqueName: \"kubernetes.io/projected/4369d13e-0dde-416a-b48e-2fec6e7a3317-kube-api-access-zskqc\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174108    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db346dd5-9969-4a80-9cd0-5c8847bf7d6e-host-slash\") pod \"iptables-alerter-9gkvt\" (UID: \"db346dd5-9969-4a80-9cd0-5c8847bf7d6e\") " pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174127    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db346dd5-9969-4a80-9cd0-5c8847bf7d6e-iptables-alerter-script\") pod \"iptables-alerter-9gkvt\" (UID: \"db346dd5-9969-4a80-9cd0-5c8847bf7d6e\") " pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174144    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-run-netns\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174152    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-kubernetes\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174178    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-run\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174215    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-cnibin\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174179    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-cnibin\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174244    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-var-lib-cni-multus\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174266    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/676d2ff5-547e-4f1e-a1ca-b87580102275-cni-binary-copy\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174282    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb485ff8-22b6-40e7-ab56-e64e94f03cbe-serviceca\") pod \"node-ca-2kt8q\" (UID: \"bb485ff8-22b6-40e7-ab56-e64e94f03cbe\") " pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174297    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-var-lib-kubelet\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174354    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-var-lib-kubelet\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174364    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-kubernetes\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174410    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-var-lib-kubelet\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174435    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-hostroot\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174433    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4369d13e-0dde-416a-b48e-2fec6e7a3317-cni-binary-copy\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174458    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-daemon-config\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174484    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-lib-modules\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.174971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174494    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-var-lib-kubelet\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174507    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-cni-dir\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174536    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-socket-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174565    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-system-cni-dir\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174579    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-hostroot\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174588    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-var-lib-cni-bin\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174615    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174639    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174662    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-socket-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174537    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-var-lib-cni-multus\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174711    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-host\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174789    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db346dd5-9969-4a80-9cd0-5c8847bf7d6e-iptables-alerter-script\") pod \"iptables-alerter-9gkvt\" (UID: \"db346dd5-9969-4a80-9cd0-5c8847bf7d6e\") " pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174800    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/676d2ff5-547e-4f1e-a1ca-b87580102275-cni-binary-copy\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174826    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-lib-modules\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174849    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-system-cni-dir\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174854    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-var-lib-cni-bin\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174877    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-cni-dir\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.175786 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174663    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-host\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174901    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174903    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174919    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-run-multus-certs\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174948    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-etc-kubernetes\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.174975    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e7bf73-8fe8-429f-b001-ed59f4e2bd01-tmp-dir\") pod \"node-resolver-8tz2q\" (UID: \"99e7bf73-8fe8-429f-b001-ed59f4e2bd01\") " pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175000    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-os-release\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175031    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-socket-dir-parent\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175058    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4tjgg\" (UniqueName: \"kubernetes.io/projected/146efc86-191d-4226-adfb-f36ee7ad4e1a-kube-api-access-4tjgg\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175106    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99e7bf73-8fe8-429f-b001-ed59f4e2bd01-hosts-file\") pod \"node-resolver-8tz2q\" (UID: \"99e7bf73-8fe8-429f-b001-ed59f4e2bd01\") " pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175109    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/676d2ff5-547e-4f1e-a1ca-b87580102275-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175149    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-sysctl-d\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175175    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-conf-dir\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175196    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/676d2ff5-547e-4f1e-a1ca-b87580102275-os-release\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175208    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8vc\" (UniqueName: \"kubernetes.io/projected/df793786-616a-4001-9c32-2e4ebd101bb2-kube-api-access-8f8vc\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175212    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-daemon-config\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175238    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-run-multus-certs\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.176601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175242    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-modprobe-d\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175273    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-etc-kubernetes\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175281    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-os-release\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175295    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-socket-dir-parent\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175307    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-run-k8s-cni-cncf-io\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175324    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-multus-conf-dir\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175362    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-device-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175394    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-systemd\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175457    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-sysctl-d\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175459    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-modprobe-d\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175516    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-host-run-k8s-cni-cncf-io\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175542    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e7bf73-8fe8-429f-b001-ed59f4e2bd01-tmp-dir\") pod \"node-resolver-8tz2q\" (UID: \"99e7bf73-8fe8-429f-b001-ed59f4e2bd01\") " pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175584    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4369d13e-0dde-416a-b48e-2fec6e7a3317-os-release\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175598    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/146efc86-191d-4226-adfb-f36ee7ad4e1a-device-dir\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175618    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99e7bf73-8fe8-429f-b001-ed59f4e2bd01-hosts-file\") pod \"node-resolver-8tz2q\" (UID: \"99e7bf73-8fe8-429f-b001-ed59f4e2bd01\") " pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175722    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-systemd\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175815    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-etc-tuned\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175848    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bb485ff8-22b6-40e7-ab56-e64e94f03cbe-serviceca\") pod \"node-ca-2kt8q\" (UID: \"bb485ff8-22b6-40e7-ab56-e64e94f03cbe\") " pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.177374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.175858    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/676d2ff5-547e-4f1e-a1ca-b87580102275-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.178269 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.176626    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-tmp\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.191961 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.191907    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkvt\" (UniqueName: \"kubernetes.io/projected/2fb76dd0-76d7-4eb0-ad36-56c371d98a7e-kube-api-access-cvkvt\") pod \"tuned-p8kkf\" (UID: \"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e\") " pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.193861 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.193843    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxj9\" (UniqueName: \"kubernetes.io/projected/db346dd5-9969-4a80-9cd0-5c8847bf7d6e-kube-api-access-4qxj9\") pod \"iptables-alerter-9gkvt\" (UID: \"db346dd5-9969-4a80-9cd0-5c8847bf7d6e\") " pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.198462 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.198435    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tjgg\" (UniqueName: \"kubernetes.io/projected/146efc86-191d-4226-adfb-f36ee7ad4e1a-kube-api-access-4tjgg\") pod \"aws-ebs-csi-driver-node-qrv2s\" (UID: \"146efc86-191d-4226-adfb-f36ee7ad4e1a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.199171 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.199107    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n8mq\" (UniqueName: \"kubernetes.io/projected/99e7bf73-8fe8-429f-b001-ed59f4e2bd01-kube-api-access-9n8mq\") pod \"node-resolver-8tz2q\" (UID: \"99e7bf73-8fe8-429f-b001-ed59f4e2bd01\") " pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.199593 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.199569    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7j65\" (UniqueName: \"kubernetes.io/projected/bb485ff8-22b6-40e7-ab56-e64e94f03cbe-kube-api-access-s7j65\") pod \"node-ca-2kt8q\" (UID: \"bb485ff8-22b6-40e7-ab56-e64e94f03cbe\") " pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.200421 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.200403    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8vc\" (UniqueName: \"kubernetes.io/projected/df793786-616a-4001-9c32-2e4ebd101bb2-kube-api-access-8f8vc\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:29:59.200667 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.200642    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzq5\" (UniqueName: \"kubernetes.io/projected/676d2ff5-547e-4f1e-a1ca-b87580102275-kube-api-access-hnzq5\") pod \"multus-additional-cni-plugins-k2kcl\" (UID: \"676d2ff5-547e-4f1e-a1ca-b87580102275\") " pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.200833 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.200818    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zskqc\" (UniqueName: \"kubernetes.io/projected/4369d13e-0dde-416a-b48e-2fec6e7a3317-kube-api-access-zskqc\") pod \"multus-978tk\" (UID: \"4369d13e-0dde-416a-b48e-2fec6e7a3317\") " pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.263808 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.263783    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:29:59.273011 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.272987    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kt8q"
Apr 24 22:29:59.284733 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.284709    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:29:59.292738 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.292717    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9gkvt"
Apr 24 22:29:59.298350 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.298326    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s"
Apr 24 22:29:59.305963 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.305937    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p8kkf"
Apr 24 22:29:59.312153 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.312137    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8tz2q"
Apr 24 22:29:59.318630 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.318613    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-978tk"
Apr 24 22:29:59.323155 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.323141    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k2kcl"
Apr 24 22:29:59.679182 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.679112    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:29:59.679182 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:29:59.679151    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:29:59.679413 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.679263    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 22:29:59.679413 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.679267    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:29:59.679413 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.679275    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 22:29:59.679413 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.679289    2543 projected.go:194] Error preparing data for projected volume kube-api-access-78bxk for pod openshift-network-diagnostics/network-check-target-2lldp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:29:59.679413 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.679354    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs podName:df793786-616a-4001-9c32-2e4ebd101bb2 nodeName:}" failed. No retries permitted until 2026-04-24 22:30:00.679318893 +0000 UTC m=+4.105716404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs") pod "network-metrics-daemon-pxq7f" (UID: "df793786-616a-4001-9c32-2e4ebd101bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:29:59.679413 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:29:59.679379    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk podName:e7d5adb7-ab2b-47df-859a-74f9de07e09b nodeName:}" failed. No retries permitted until 2026-04-24 22:30:00.679364303 +0000 UTC m=+4.105761814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bxk" (UniqueName: "kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk") pod "network-check-target-2lldp" (UID: "e7d5adb7-ab2b-47df-859a-74f9de07e09b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:29:59.704664 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:59.704642    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb76dd0_76d7_4eb0_ad36_56c371d98a7e.slice/crio-7811bd7261f75d05afdeb825f0ebc0e314788523e2c5f9bb9024b19327cee875 WatchSource:0}: Error finding container 7811bd7261f75d05afdeb825f0ebc0e314788523e2c5f9bb9024b19327cee875: Status 404 returned error can't find the container with id 7811bd7261f75d05afdeb825f0ebc0e314788523e2c5f9bb9024b19327cee875
Apr 24 22:29:59.705634 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:59.705612    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7de9be_e366_4f28_8f32_0587de85e41b.slice/crio-db9e8b3b7dec5bc957267ea30019b0d6a08a455bfeaee98fe61fd4d34af11456 WatchSource:0}: Error finding container db9e8b3b7dec5bc957267ea30019b0d6a08a455bfeaee98fe61fd4d34af11456: Status 404 returned error can't find the container with id db9e8b3b7dec5bc957267ea30019b0d6a08a455bfeaee98fe61fd4d34af11456
Apr 24 22:29:59.706964 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:29:59.706946    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode170e736_789d_406e_a238_bcaa098a2309.slice/crio-5d9a58ceac39a98cbd89da4b2d695acf45e5600833f9da8f35b6d22c389c272e WatchSource:0}: Error finding container 5d9a58ceac39a98cbd89da4b2d695acf45e5600833f9da8f35b6d22c389c272e: Status 404 returned error can't find the container with id 5d9a58ceac39a98cbd89da4b2d695acf45e5600833f9da8f35b6d22c389c272e
Apr 24 22:30:00.008192 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.007937    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-23 22:24:57 +0000 UTC" deadline="2028-01-09 18:49:11.420056826 +0000 UTC"
Apr 24 22:30:00.008192 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.008121    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14996h19m11.411940088s"
Apr 24 22:30:00.059090 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.058328    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kt8q" event={"ID":"bb485ff8-22b6-40e7-ab56-e64e94f03cbe","Type":"ContainerStarted","Data":"5a995eb6b7c4fbc49c22a1405d7a6a5ac118b2d8cd62c384ff85ceddef0dba75"}
Apr 24 22:30:00.059759 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.059676    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2kcl" event={"ID":"676d2ff5-547e-4f1e-a1ca-b87580102275","Type":"ContainerStarted","Data":"789e3d052a0e6badaa569e9ab9a0011780f91e6ed98eebd9ee17481915670a9e"}
Apr 24 22:30:00.060988 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.060961    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkmd6" event={"ID":"e170e736-789d-406e-a238-bcaa098a2309","Type":"ContainerStarted","Data":"5d9a58ceac39a98cbd89da4b2d695acf45e5600833f9da8f35b6d22c389c272e"}
Apr 24 22:30:00.062698 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.062666    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerStarted","Data":"db9e8b3b7dec5bc957267ea30019b0d6a08a455bfeaee98fe61fd4d34af11456"}
Apr 24 22:30:00.065082 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.065037    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p8kkf" event={"ID":"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e","Type":"ContainerStarted","Data":"7811bd7261f75d05afdeb825f0ebc0e314788523e2c5f9bb9024b19327cee875"}
Apr 24 22:30:00.068456 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.068431    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal" event={"ID":"60d03c02f9ed5519c8c0179698b80bb4","Type":"ContainerStarted","Data":"eb5efbc5726d5eec68fc3b8ec932b0117f667bbf9a506e036bc29fddc60c5afd"}
Apr 24 22:30:00.072857 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.072834    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9gkvt" event={"ID":"db346dd5-9969-4a80-9cd0-5c8847bf7d6e","Type":"ContainerStarted","Data":"31d407d78cdc7386c3a52da4ad467d57a42ed7847d7745e7ac8a49ca4f571a1b"}
Apr 24 22:30:00.075232 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.075199    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s" event={"ID":"146efc86-191d-4226-adfb-f36ee7ad4e1a","Type":"ContainerStarted","Data":"01f2fd974703b6cde37a0aa05e3c8d0a459b5aa3a5113c3af01b2c0d877b11a5"}
Apr 24 22:30:00.078088 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.078013    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-978tk" event={"ID":"4369d13e-0dde-416a-b48e-2fec6e7a3317","Type":"ContainerStarted","Data":"93352c9a3844738fe02a2016b925758d78ec6a3c7b36aa5e8a37778495514f4f"}
Apr 24 22:30:00.082374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.082318    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8tz2q" event={"ID":"99e7bf73-8fe8-429f-b001-ed59f4e2bd01","Type":"ContainerStarted","Data":"285a63f26689092e3c57e061e2f724c06d0e76a14ec6edc1961fe7133e99a317"}
Apr 24 22:30:00.687732 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.687649    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:00.687882 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.687737    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:00.687882 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:00.687851    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:00.688000 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:00.687911    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs podName:df793786-616a-4001-9c32-2e4ebd101bb2 nodeName:}" failed. No retries permitted until 2026-04-24 22:30:02.687893837 +0000 UTC m=+6.114291358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs") pod "network-metrics-daemon-pxq7f" (UID: "df793786-616a-4001-9c32-2e4ebd101bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:00.688364 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:00.688325    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 22:30:00.688364 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:00.688370    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 22:30:00.688500 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:00.688384    2543 projected.go:194] Error preparing data for projected volume kube-api-access-78bxk for pod openshift-network-diagnostics/network-check-target-2lldp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:00.688500 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:00.688428    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk podName:e7d5adb7-ab2b-47df-859a-74f9de07e09b nodeName:}" failed. No retries permitted until 2026-04-24 22:30:02.688412664 +0000 UTC m=+6.114810229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bxk" (UniqueName: "kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk") pod "network-check-target-2lldp" (UID: "e7d5adb7-ab2b-47df-859a-74f9de07e09b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:00.699842 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:00.699710    2543 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 24 22:30:01.051554 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:01.051486    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:01.051965 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:01.051489    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:01.051965 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:01.051597    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:01.051965 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:01.051696    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:01.098934 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:01.098903    2543 generic.go:358] "Generic (PLEG): container finished" podID="de4cd73d2cc50d777b3fcaab17c0aa50" containerID="25c8dfb12a19af687411ac0cb1639c98ff7db6c4024f77344e0eb4ee5cd5c041" exitCode=0
Apr 24 22:30:01.099082 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:01.098982    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal" event={"ID":"de4cd73d2cc50d777b3fcaab17c0aa50","Type":"ContainerDied","Data":"25c8dfb12a19af687411ac0cb1639c98ff7db6c4024f77344e0eb4ee5cd5c041"}
Apr 24 22:30:01.115901 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:01.115841    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal" podStartSLOduration=3.115824577 podStartE2EDuration="3.115824577s" podCreationTimestamp="2026-04-24 22:29:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:30:00.090660012 +0000 UTC m=+3.517057542" watchObservedRunningTime="2026-04-24 22:30:01.115824577 +0000 UTC m=+4.542222100"
Apr 24 22:30:02.112287 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:02.112248    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal" event={"ID":"de4cd73d2cc50d777b3fcaab17c0aa50","Type":"ContainerStarted","Data":"3660349688f463335562cf28b4ba0231230f867f1da5c5dba258c19a5cbb062c"}
Apr 24 22:30:02.147175 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:02.146679    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal" podStartSLOduration=4.146659365 podStartE2EDuration="4.146659365s" podCreationTimestamp="2026-04-24 22:29:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:30:02.146019944 +0000 UTC m=+5.572417476" watchObservedRunningTime="2026-04-24 22:30:02.146659365 +0000 UTC m=+5.573057011"
Apr 24 22:30:02.707887 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:02.707847    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:02.708043 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:02.707917    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:02.708095 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:02.708069    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 22:30:02.708133 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:02.708094    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 22:30:02.708133 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:02.708108    2543 projected.go:194] Error preparing data for projected volume kube-api-access-78bxk for pod openshift-network-diagnostics/network-check-target-2lldp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:02.708187 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:02.708165    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk podName:e7d5adb7-ab2b-47df-859a-74f9de07e09b nodeName:}" failed. No retries permitted until 2026-04-24 22:30:06.708147254 +0000 UTC m=+10.134544765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bxk" (UniqueName: "kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk") pod "network-check-target-2lldp" (UID: "e7d5adb7-ab2b-47df-859a-74f9de07e09b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:02.708617 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:02.708595    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:02.708681 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:02.708653    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs podName:df793786-616a-4001-9c32-2e4ebd101bb2 nodeName:}" failed. No retries permitted until 2026-04-24 22:30:06.708637527 +0000 UTC m=+10.135035039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs") pod "network-metrics-daemon-pxq7f" (UID: "df793786-616a-4001-9c32-2e4ebd101bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:03.053568 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:03.053491    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:03.053735 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:03.053627    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:03.054432 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:03.053491    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:03.054432 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:03.053962    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:05.052098 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:05.051642    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:05.052098 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:05.051760    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:05.052098 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:05.051997    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:05.052685 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:05.052106    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:06.740516 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:06.740479    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:06.740853 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:06.740538    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:06.740853 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:06.740684    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 22:30:06.740853 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:06.740737    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 22:30:06.740853 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:06.740750    2543 projected.go:194] Error preparing data for projected volume kube-api-access-78bxk for pod openshift-network-diagnostics/network-check-target-2lldp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:06.740853 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:06.740807    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk podName:e7d5adb7-ab2b-47df-859a-74f9de07e09b nodeName:}" failed. No retries permitted until 2026-04-24 22:30:14.740789563 +0000 UTC m=+18.167187072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bxk" (UniqueName: "kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk") pod "network-check-target-2lldp" (UID: "e7d5adb7-ab2b-47df-859a-74f9de07e09b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:06.741297 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:06.741279    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:06.741412 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:06.741325    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs podName:df793786-616a-4001-9c32-2e4ebd101bb2 nodeName:}" failed. No retries permitted until 2026-04-24 22:30:14.741310836 +0000 UTC m=+18.167708344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs") pod "network-metrics-daemon-pxq7f" (UID: "df793786-616a-4001-9c32-2e4ebd101bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:07.054918 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:07.054842    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:07.055099 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:07.054972    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:07.055452 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:07.055431    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:07.055678 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:07.055653    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:09.012878 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.012842    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kcq9z"]
Apr 24 22:30:09.016312 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.015940    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.016312 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:09.016021    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:09.054386 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.053674    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:09.054386 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:09.053804    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:09.054386 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.054153    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:09.054386 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:09.054239    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:09.059578 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.059552    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d17a86f7-36e1-4352-92cf-748b8c466f7e-kubelet-config\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.059677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.059600    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.059743 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.059687    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d17a86f7-36e1-4352-92cf-748b8c466f7e-dbus\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.161025 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.160956    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d17a86f7-36e1-4352-92cf-748b8c466f7e-dbus\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.161025 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.161015    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d17a86f7-36e1-4352-92cf-748b8c466f7e-kubelet-config\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.161208 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.161093    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d17a86f7-36e1-4352-92cf-748b8c466f7e-kubelet-config\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.161208 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.161136    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.161208 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.161170    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d17a86f7-36e1-4352-92cf-748b8c466f7e-dbus\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.161371 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:09.161283    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:09.161371 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:09.161360    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret podName:d17a86f7-36e1-4352-92cf-748b8c466f7e nodeName:}" failed. No retries permitted until 2026-04-24 22:30:09.661323763 +0000 UTC m=+13.087721289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret") pod "global-pull-secret-syncer-kcq9z" (UID: "d17a86f7-36e1-4352-92cf-748b8c466f7e") : object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:09.664049 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:09.664017    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:09.664229 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:09.664170    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:09.664286 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:09.664239    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret podName:d17a86f7-36e1-4352-92cf-748b8c466f7e nodeName:}" failed. No retries permitted until 2026-04-24 22:30:10.66421859 +0000 UTC m=+14.090616117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret") pod "global-pull-secret-syncer-kcq9z" (UID: "d17a86f7-36e1-4352-92cf-748b8c466f7e") : object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:10.673507 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:10.673462    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:10.673919 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:10.673604    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:10.673919 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:10.673662    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret podName:d17a86f7-36e1-4352-92cf-748b8c466f7e nodeName:}" failed. No retries permitted until 2026-04-24 22:30:12.673647426 +0000 UTC m=+16.100044934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret") pod "global-pull-secret-syncer-kcq9z" (UID: "d17a86f7-36e1-4352-92cf-748b8c466f7e") : object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:11.054537 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:11.054458    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:11.054537 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:11.054493    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:11.054740 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:11.054586    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:11.054740 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:11.054618    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:11.054740 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:11.054729    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:11.054878 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:11.054787    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:12.689043 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:12.688951    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:12.689471 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:12.689091    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:12.689471 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:12.689166    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret podName:d17a86f7-36e1-4352-92cf-748b8c466f7e nodeName:}" failed. No retries permitted until 2026-04-24 22:30:16.689146391 +0000 UTC m=+20.115543913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret") pod "global-pull-secret-syncer-kcq9z" (UID: "d17a86f7-36e1-4352-92cf-748b8c466f7e") : object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:13.051053 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:13.050972    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:13.051202 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:13.050971    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:13.051202 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:13.050971    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:13.051202 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:13.051175    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:13.051412 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:13.051106    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:13.051412 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:13.051217    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:14.809007 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:14.808960    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:14.809407 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:14.809043    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:14.809407 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:14.809125    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:14.809407 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:14.809136    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 22:30:14.809407 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:14.809159    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 22:30:14.809407 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:14.809173    2543 projected.go:194] Error preparing data for projected volume kube-api-access-78bxk for pod openshift-network-diagnostics/network-check-target-2lldp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:14.809407 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:14.809176    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs podName:df793786-616a-4001-9c32-2e4ebd101bb2 nodeName:}" failed. No retries permitted until 2026-04-24 22:30:30.809162986 +0000 UTC m=+34.235560494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs") pod "network-metrics-daemon-pxq7f" (UID: "df793786-616a-4001-9c32-2e4ebd101bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:14.809407 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:14.809223    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk podName:e7d5adb7-ab2b-47df-859a-74f9de07e09b nodeName:}" failed. No retries permitted until 2026-04-24 22:30:30.809207912 +0000 UTC m=+34.235605435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bxk" (UniqueName: "kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk") pod "network-check-target-2lldp" (UID: "e7d5adb7-ab2b-47df-859a-74f9de07e09b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:15.051072 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:15.051040    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:15.051072 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:15.051074    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:15.051294 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:15.051116    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:15.051294 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:15.051212    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:15.051425 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:15.051358    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:15.051489 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:15.051470    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:16.723490 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:16.723450    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:16.723834 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:16.723596    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:16.723834 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:16.723656    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret podName:d17a86f7-36e1-4352-92cf-748b8c466f7e nodeName:}" failed. No retries permitted until 2026-04-24 22:30:24.723642053 +0000 UTC m=+28.150039561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret") pod "global-pull-secret-syncer-kcq9z" (UID: "d17a86f7-36e1-4352-92cf-748b8c466f7e") : object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:17.051806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:17.051771    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:17.051945 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:17.051850    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:17.051945 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:17.051934    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:17.052059 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:17.052039    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:17.052124 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:17.052090    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:17.052187 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:17.052169    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:18.139941 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.139571    2543 generic.go:358] "Generic (PLEG): container finished" podID="676d2ff5-547e-4f1e-a1ca-b87580102275" containerID="5bbca5c6fbc6073b2a2ae8253e38e79e1ac7b045a471435180481a751612adfd" exitCode=0
Apr 24 22:30:18.139941 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.139674    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2kcl" event={"ID":"676d2ff5-547e-4f1e-a1ca-b87580102275","Type":"ContainerDied","Data":"5bbca5c6fbc6073b2a2ae8253e38e79e1ac7b045a471435180481a751612adfd"}
Apr 24 22:30:18.141756 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.141730    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkmd6" event={"ID":"e170e736-789d-406e-a238-bcaa098a2309","Type":"ContainerStarted","Data":"8a8bda59849c45b7a5f91af5cabcfb53a1600ee55849c6a3413f5c4be76bcbed"}
Apr 24 22:30:18.144617 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.144597    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:30:18.145068 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.145045    2543 generic.go:358] "Generic (PLEG): container finished" podID="8b7de9be-e366-4f28-8f32-0587de85e41b" containerID="085be3bb9af48c5f4a1d879ad9cd5c497d356b8c1ee9d658f8a007e55ed6cf6e" exitCode=1
Apr 24 22:30:18.145194 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.145092    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerStarted","Data":"655c28fb1999cdeefbb154a435313197bafedb454384a74e8d6810e4c53c8988"}
Apr 24 22:30:18.145194 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.145133    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerStarted","Data":"55472f9e5b559b3d98bf085eca9d548b7738da9d3c04cb6cb420bd6ce5cb2f20"}
Apr 24 22:30:18.145194 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.145149    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerStarted","Data":"4474ab6e8cdeeeba0ddc28ea135782092dfb3fad7f35bb9da4f561fa12725796"}
Apr 24 22:30:18.145194 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.145157    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerStarted","Data":"7c31246730e577654b3a56a744d8f318ad75df947bed9db804b3e3a3fdfd8d6c"}
Apr 24 22:30:18.145194 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.145164    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerDied","Data":"085be3bb9af48c5f4a1d879ad9cd5c497d356b8c1ee9d658f8a007e55ed6cf6e"}
Apr 24 22:30:18.145194 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.145173    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerStarted","Data":"bacac10f2081361fc92c938283b0f15beda09b958d21ba6cc8fba269f93190ff"}
Apr 24 22:30:18.146680 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.146649    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p8kkf" event={"ID":"2fb76dd0-76d7-4eb0-ad36-56c371d98a7e","Type":"ContainerStarted","Data":"54c3cd3bef32cccab1b54f9ccc68a793317d7b92e26d742cc6d5e124c52ebc5c"}
Apr 24 22:30:18.148089 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.148067    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s" event={"ID":"146efc86-191d-4226-adfb-f36ee7ad4e1a","Type":"ContainerStarted","Data":"431e921931ffb4d6b9c0628b98b512a7a90bad154c7a5c6bd62679a8a0a2b67e"}
Apr 24 22:30:18.149595 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.149568    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-978tk" event={"ID":"4369d13e-0dde-416a-b48e-2fec6e7a3317","Type":"ContainerStarted","Data":"4de904b8f049e133b0f1a3b2fca3e8bf6f6d66610072aa12e8be59c596eceb87"}
Apr 24 22:30:18.150930 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.150890    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8tz2q" event={"ID":"99e7bf73-8fe8-429f-b001-ed59f4e2bd01","Type":"ContainerStarted","Data":"71d44f73ba7fd63584306c0a037cdb81dc9968e7a86e662e73acc57f4b89811d"}
Apr 24 22:30:18.152240 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.152218    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kt8q" event={"ID":"bb485ff8-22b6-40e7-ab56-e64e94f03cbe","Type":"ContainerStarted","Data":"89ea78308336e488196bdd56929def5955ad1b3b445eda3abfbd1c165067a2e9"}
Apr 24 22:30:18.222475 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.222418    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2kt8q" podStartSLOduration=3.764591652 podStartE2EDuration="21.222403123s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:29:59.718661607 +0000 UTC m=+3.145059127" lastFinishedPulling="2026-04-24 22:30:17.176473087 +0000 UTC m=+20.602870598" observedRunningTime="2026-04-24 22:30:18.221759127 +0000 UTC m=+21.648156656" watchObservedRunningTime="2026-04-24 22:30:18.222403123 +0000 UTC m=+21.648800657"
Apr 24 22:30:18.246834 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.246794    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-8tz2q" podStartSLOduration=3.735674028 podStartE2EDuration="21.246780981s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:29:59.711672179 +0000 UTC m=+3.138069688" lastFinishedPulling="2026-04-24 22:30:17.22277913 +0000 UTC m=+20.649176641" observedRunningTime="2026-04-24 22:30:18.246781859 +0000 UTC m=+21.673179388" watchObservedRunningTime="2026-04-24 22:30:18.246780981 +0000 UTC m=+21.673178511"
Apr 24 22:30:18.297567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.297533    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-978tk" podStartSLOduration=3.773508498 podStartE2EDuration="21.297522659s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:29:59.718562212 +0000 UTC m=+3.144959735" lastFinishedPulling="2026-04-24 22:30:17.242576389 +0000 UTC m=+20.668973896" observedRunningTime="2026-04-24 22:30:18.27198926 +0000 UTC m=+21.698386791" watchObservedRunningTime="2026-04-24 22:30:18.297522659 +0000 UTC m=+21.723920189"
Apr 24 22:30:18.297746 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.297726    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zkmd6" podStartSLOduration=8.702235005 podStartE2EDuration="21.297722576s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:29:59.710549827 +0000 UTC m=+3.136947339" lastFinishedPulling="2026-04-24 22:30:12.306037396 +0000 UTC m=+15.732434910" observedRunningTime="2026-04-24 22:30:18.297639533 +0000 UTC m=+21.724037041" watchObservedRunningTime="2026-04-24 22:30:18.297722576 +0000 UTC m=+21.724120105"
Apr 24 22:30:18.334920 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.334889    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p8kkf" podStartSLOduration=3.819636081 podStartE2EDuration="21.334878263s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:29:59.706431282 +0000 UTC m=+3.132828804" lastFinishedPulling="2026-04-24 22:30:17.221673463 +0000 UTC m=+20.648070986" observedRunningTime="2026-04-24 22:30:18.334602927 +0000 UTC m=+21.761000468" watchObservedRunningTime="2026-04-24 22:30:18.334878263 +0000 UTC m=+21.761275793"
Apr 24 22:30:18.565416 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:18.565392    2543 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Apr 24 22:30:19.040111 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:19.039969    2543 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-24T22:30:18.565412172Z","UUID":"79dcbb52-7ad3-4dcc-bced-fd98f544e949","Handler":null,"Name":"","Endpoint":""}
Apr 24 22:30:19.042441 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:19.042405    2543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Apr 24 22:30:19.042441 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:19.042438    2543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Apr 24 22:30:19.054500 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:19.053761    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:19.054500 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:19.053867    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:19.054500 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:19.054209    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:19.054500 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:19.054305    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:19.054500 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:19.054393    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:19.054500 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:19.054459    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:19.156215 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:19.156182    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9gkvt" event={"ID":"db346dd5-9969-4a80-9cd0-5c8847bf7d6e","Type":"ContainerStarted","Data":"e47eba3c0099eb86a744481f170961ad2acf44439a0d271da09fa444386194d6"}
Apr 24 22:30:19.158411 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:19.158379    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s" event={"ID":"146efc86-191d-4226-adfb-f36ee7ad4e1a","Type":"ContainerStarted","Data":"471050a2eac881993d14a78816c9969b4439b1202ebc43ddd03e19d24077169d"}
Apr 24 22:30:20.161968 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:20.161791    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s" event={"ID":"146efc86-191d-4226-adfb-f36ee7ad4e1a","Type":"ContainerStarted","Data":"deb5c80ce4e6883aa27eea5ed831bd2cfdcb72262d111db15a7f0e87f3bfcdb3"}
Apr 24 22:30:20.184079 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:20.184054    2543 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:30:20.184589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:20.184572    2543 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:30:20.189566 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:20.189526    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9gkvt" podStartSLOduration=5.6872518450000005 podStartE2EDuration="23.189511808s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:29:59.7187025 +0000 UTC m=+3.145100015" lastFinishedPulling="2026-04-24 22:30:17.220962464 +0000 UTC m=+20.647359978" observedRunningTime="2026-04-24 22:30:19.201383773 +0000 UTC m=+22.627781303" watchObservedRunningTime="2026-04-24 22:30:20.189511808 +0000 UTC m=+23.615909342"
Apr 24 22:30:20.216092 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:20.216040    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qrv2s" podStartSLOduration=3.461773108 podStartE2EDuration="23.216027074s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:29:59.718530945 +0000 UTC m=+3.144928466" lastFinishedPulling="2026-04-24 22:30:19.47278492 +0000 UTC m=+22.899182432" observedRunningTime="2026-04-24 22:30:20.19068892 +0000 UTC m=+23.617086452" watchObservedRunningTime="2026-04-24 22:30:20.216027074 +0000 UTC m=+23.642424605"
Apr 24 22:30:21.050938 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:21.050901    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:21.051123 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:21.051004    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:21.051123 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:21.051033    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:21.051123 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:21.051000    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:21.051123 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:21.051094    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:21.051310 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:21.051171    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:21.166262 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:21.166239    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:30:21.167012 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:21.166978    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerStarted","Data":"6fb45bfcfbba5a2cafe352698592846c9a4cc5e54255308a3d0f787b8b87393b"}
Apr 24 22:30:21.871815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:21.871781    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:30:21.872449 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:21.872426    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zkmd6"
Apr 24 22:30:23.051280 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.051116    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:23.051899 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.051129    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:23.051899 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:23.051359    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:23.051899 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.051144    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:23.051899 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:23.051424    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:23.051899 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:23.051524    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:23.144953 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.144933    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8tz2q_99e7bf73-8fe8-429f-b001-ed59f4e2bd01/dns-node-resolver/0.log"
Apr 24 22:30:23.171741 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.171717    2543 generic.go:358] "Generic (PLEG): container finished" podID="676d2ff5-547e-4f1e-a1ca-b87580102275" containerID="beb032f84a558dbb15634a973970e3a6dc68275bd283591e350efeebee5f1873" exitCode=0
Apr 24 22:30:23.171861 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.171798    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2kcl" event={"ID":"676d2ff5-547e-4f1e-a1ca-b87580102275","Type":"ContainerDied","Data":"beb032f84a558dbb15634a973970e3a6dc68275bd283591e350efeebee5f1873"}
Apr 24 22:30:23.174917 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.174901    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:30:23.175252 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.175230    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerStarted","Data":"afa6a63d5fcf27e2f5a0c3dcdcf1246e81276381f92bbf286ec127e9ef95127f"}
Apr 24 22:30:23.175509 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.175455    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:30:23.175658 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.175639    2543 scope.go:117] "RemoveContainer" containerID="085be3bb9af48c5f4a1d879ad9cd5c497d356b8c1ee9d658f8a007e55ed6cf6e"
Apr 24 22:30:23.191538 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:23.191522    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:30:24.110610 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.110553    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2kt8q_bb485ff8-22b6-40e7-ab56-e64e94f03cbe/node-ca/0.log"
Apr 24 22:30:24.179741 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.179685    2543 generic.go:358] "Generic (PLEG): container finished" podID="676d2ff5-547e-4f1e-a1ca-b87580102275" containerID="5bdf4e6a6dadfb4a60c4a8ab554eeb5441e89612317eb3fc2758e77cb79a7eca" exitCode=0
Apr 24 22:30:24.179872 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.179805    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2kcl" event={"ID":"676d2ff5-547e-4f1e-a1ca-b87580102275","Type":"ContainerDied","Data":"5bdf4e6a6dadfb4a60c4a8ab554eeb5441e89612317eb3fc2758e77cb79a7eca"}
Apr 24 22:30:24.183775 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.183754    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:30:24.184084 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.184061    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" event={"ID":"8b7de9be-e366-4f28-8f32-0587de85e41b","Type":"ContainerStarted","Data":"936d8ae7e9cb0eb4d772884963d2dd00ffc385b446e6f3902d9f51b546cbcd5c"}
Apr 24 22:30:24.184353 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.184319    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:30:24.184442 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.184359    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:30:24.199833 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.199810    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:30:24.455167 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.454947    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2" podStartSLOduration=9.887179188 podStartE2EDuration="27.454932109s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:29:59.707292534 +0000 UTC m=+3.133690042" lastFinishedPulling="2026-04-24 22:30:17.275045446 +0000 UTC m=+20.701442963" observedRunningTime="2026-04-24 22:30:24.299156907 +0000 UTC m=+27.725554437" watchObservedRunningTime="2026-04-24 22:30:24.454932109 +0000 UTC m=+27.881329641"
Apr 24 22:30:24.455308 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.455265    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kcq9z"]
Apr 24 22:30:24.455374 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.455366    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:24.455459 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:24.455443    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:24.462947 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.462913    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2lldp"]
Apr 24 22:30:24.463066 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.463025    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:24.463132 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:24.463102    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:24.463600 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.463574    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pxq7f"]
Apr 24 22:30:24.463711 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.463686    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:24.463803 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:24.463779    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:24.785900 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:24.785835    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:24.786057 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:24.785944    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:24.786057 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:24.785990    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret podName:d17a86f7-36e1-4352-92cf-748b8c466f7e nodeName:}" failed. No retries permitted until 2026-04-24 22:30:40.785977629 +0000 UTC m=+44.212375137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret") pod "global-pull-secret-syncer-kcq9z" (UID: "d17a86f7-36e1-4352-92cf-748b8c466f7e") : object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:25.187306 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:25.187228    2543 generic.go:358] "Generic (PLEG): container finished" podID="676d2ff5-547e-4f1e-a1ca-b87580102275" containerID="4cdc97b70d538c5d71ed2b7713a11aa77dba572628cf137e65ede5f4e030f0df" exitCode=0
Apr 24 22:30:25.187841 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:25.187299    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2kcl" event={"ID":"676d2ff5-547e-4f1e-a1ca-b87580102275","Type":"ContainerDied","Data":"4cdc97b70d538c5d71ed2b7713a11aa77dba572628cf137e65ede5f4e030f0df"}
Apr 24 22:30:26.051672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:26.051638    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:26.051846 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:26.051732    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:26.051846 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:26.051638    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:26.051846 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:26.051765    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:26.051967 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:26.051847    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:26.051967 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:26.051922    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:28.051307 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:28.051271    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:28.051307 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:28.051290    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:28.052010 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:28.051283    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:28.052010 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:28.051432    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:28.052010 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:28.051550    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:28.052010 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:28.051615    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:30.051488 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:30.051449    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:30.052141 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:30.051488    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:30.052141 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:30.051451    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:30.052141 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:30.051589    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:30.052141 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:30.051660    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:30.052141 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:30.051722    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:30.841477 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:30.841445    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:30.841657 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:30.841489    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:30.841657 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:30.841584    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 24 22:30:30.841657 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:30.841595    2543 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 24 22:30:30.841657 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:30.841593    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:30.841657 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:30.841604    2543 projected.go:194] Error preparing data for projected volume kube-api-access-78bxk for pod openshift-network-diagnostics/network-check-target-2lldp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:30.841657 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:30.841650    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk podName:e7d5adb7-ab2b-47df-859a-74f9de07e09b nodeName:}" failed. No retries permitted until 2026-04-24 22:31:02.841638469 +0000 UTC m=+66.268035977 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-78bxk" (UniqueName: "kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk") pod "network-check-target-2lldp" (UID: "e7d5adb7-ab2b-47df-859a-74f9de07e09b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 24 22:30:30.841885 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:30.841663    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs podName:df793786-616a-4001-9c32-2e4ebd101bb2 nodeName:}" failed. No retries permitted until 2026-04-24 22:31:02.84165683 +0000 UTC m=+66.268054338 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs") pod "network-metrics-daemon-pxq7f" (UID: "df793786-616a-4001-9c32-2e4ebd101bb2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 24 22:30:32.051578 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:32.051545    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:32.051578 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:32.051576    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:32.051944 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:32.051584    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:32.051944 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:32.051668    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:32.051944 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:32.051729    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:32.051944 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:32.051774    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:32.202854 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:32.202825    2543 generic.go:358] "Generic (PLEG): container finished" podID="676d2ff5-547e-4f1e-a1ca-b87580102275" containerID="8e8163e4090bd3994e0200e88df49d08e2825c8a377a3281d5d2b23835b5cd91" exitCode=0
Apr 24 22:30:32.202998 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:32.202885    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2kcl" event={"ID":"676d2ff5-547e-4f1e-a1ca-b87580102275","Type":"ContainerDied","Data":"8e8163e4090bd3994e0200e88df49d08e2825c8a377a3281d5d2b23835b5cd91"}
Apr 24 22:30:33.207371 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:33.207177    2543 generic.go:358] "Generic (PLEG): container finished" podID="676d2ff5-547e-4f1e-a1ca-b87580102275" containerID="a5605c76c8193060c5979ca25c82c1af0616aa0b1d33752e957ae8cc46b83b0c" exitCode=0
Apr 24 22:30:33.207762 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:33.207251    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2kcl" event={"ID":"676d2ff5-547e-4f1e-a1ca-b87580102275","Type":"ContainerDied","Data":"a5605c76c8193060c5979ca25c82c1af0616aa0b1d33752e957ae8cc46b83b0c"}
Apr 24 22:30:34.051534 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:34.051503    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:34.051662 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:34.051553    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:34.051662 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:34.051503    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:34.051662 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:34.051623    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:34.051766 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:34.051662    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:34.051766 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:34.051730    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:34.212232 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:34.212206    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2kcl" event={"ID":"676d2ff5-547e-4f1e-a1ca-b87580102275","Type":"ContainerStarted","Data":"e453544cca33a76e73c29cd50a5cd54954573b90aa7d70d9ae19f926b8312ba6"}
Apr 24 22:30:34.241917 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:34.241878    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k2kcl" podStartSLOduration=5.854079223 podStartE2EDuration="37.241865415s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:29:59.709655659 +0000 UTC m=+3.136053186" lastFinishedPulling="2026-04-24 22:30:31.09744187 +0000 UTC m=+34.523839378" observedRunningTime="2026-04-24 22:30:34.240014375 +0000 UTC m=+37.666411905" watchObservedRunningTime="2026-04-24 22:30:34.241865415 +0000 UTC m=+37.668262945"
Apr 24 22:30:36.051425 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:36.051400    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:36.051764 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:36.051407    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:36.051764 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:36.051411    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:36.051764 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:36.051584    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:36.051764 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:36.051495    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:36.051764 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:36.051668    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:38.051565 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:38.051524    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:38.051940 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:38.051532    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:38.051940 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:38.051707    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:38.051940 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:38.051626    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:38.051940 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:38.051532    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:38.051940 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:38.051785    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:40.051696 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:40.051659    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:40.052145 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:40.051769    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:40.052145 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:40.051790    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:40.052145 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:40.051783    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:40.052145 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:40.051844    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:40.052145 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:40.051924    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:40.817718 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:40.817689    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:40.817878 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:40.817787    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:40.817878 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:40.817832    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret podName:d17a86f7-36e1-4352-92cf-748b8c466f7e nodeName:}" failed. No retries permitted until 2026-04-24 22:31:12.817819827 +0000 UTC m=+76.244217335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret") pod "global-pull-secret-syncer-kcq9z" (UID: "d17a86f7-36e1-4352-92cf-748b8c466f7e") : object "kube-system"/"original-pull-secret" not registered
Apr 24 22:30:42.051568 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.051544    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:42.051953 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.051572    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:42.051953 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.051543    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:42.051953 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:42.051682    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxq7f" podUID="df793786-616a-4001-9c32-2e4ebd101bb2"
Apr 24 22:30:42.051953 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:42.051803    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kcq9z" podUID="d17a86f7-36e1-4352-92cf-748b8c466f7e"
Apr 24 22:30:42.051953 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:42.051866    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lldp" podUID="e7d5adb7-ab2b-47df-859a-74f9de07e09b"
Apr 24 22:30:42.423980 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.423908    2543 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeReady"
Apr 24 22:30:42.424123 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.424022    2543 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Apr 24 22:30:42.471964 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.471936    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-879bc8cd9-gm7gg"]
Apr 24 22:30:42.510449 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.510426    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lp9hc"]
Apr 24 22:30:42.510589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.510573    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.520635 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.520618    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Apr 24 22:30:42.521179 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.521165    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8jvtj\""
Apr 24 22:30:42.521699 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.521683    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Apr 24 22:30:42.525549 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.525519    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Apr 24 22:30:42.532178 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.532155    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Apr 24 22:30:42.534073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.534052    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-879bc8cd9-gm7gg"]
Apr 24 22:30:42.534316 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.534090    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bs42h"]
Apr 24 22:30:42.534868 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.534850    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.537240 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.537223    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Apr 24 22:30:42.537399 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.537278    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Apr 24 22:30:42.539094 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.539080    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Apr 24 22:30:42.539155 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.539095    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Apr 24 22:30:42.540092 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.540078    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ksx55\""
Apr 24 22:30:42.559565 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.559545    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lp9hc"]
Apr 24 22:30:42.559688 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.559570    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bs42h"]
Apr 24 22:30:42.559688 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.559647    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bs42h"
Apr 24 22:30:42.565774 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.565739    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g8k4l\""
Apr 24 22:30:42.565864 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.565804    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Apr 24 22:30:42.565864 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.565840    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Apr 24 22:30:42.566014 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.565999    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Apr 24 22:30:42.609575 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.609551    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9gfb7"]
Apr 24 22:30:42.621821 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.621803    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.625037 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.625021    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hjbjm\""
Apr 24 22:30:42.625177 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.625163    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Apr 24 22:30:42.625640 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.625626    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Apr 24 22:30:42.634686 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.634667    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9gfb7"]
Apr 24 22:30:42.634853 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.634837    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-crio-socket\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.634929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.634867    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.634929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.634896    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.634929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.634923    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnplx\" (UniqueName: \"kubernetes.io/projected/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-kube-api-access-pnplx\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.635074 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.634997    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-registry-tls\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.635074 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635034    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-bound-sa-token\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.635177 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635073    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-registry-certificates\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.635177 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635099    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49fzr\" (UniqueName: \"kubernetes.io/projected/05906a72-1fb8-49fc-86c9-01d5fa92a7cd-kube-api-access-49fzr\") pod \"ingress-canary-bs42h\" (UID: \"05906a72-1fb8-49fc-86c9-01d5fa92a7cd\") " pod="openshift-ingress-canary/ingress-canary-bs42h"
Apr 24 22:30:42.635177 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635123    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-data-volume\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.635177 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635153    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjzmh\" (UniqueName: \"kubernetes.io/projected/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-kube-api-access-fjzmh\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.635322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635185    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-installation-pull-secrets\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.635322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635218    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-ca-trust-extracted\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.635322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635273    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-image-registry-private-configuration\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.635322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635314    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-trusted-ca\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.635507 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.635357    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05906a72-1fb8-49fc-86c9-01d5fa92a7cd-cert\") pod \"ingress-canary-bs42h\" (UID: \"05906a72-1fb8-49fc-86c9-01d5fa92a7cd\") " pod="openshift-ingress-canary/ingress-canary-bs42h"
Apr 24 22:30:42.735695 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735671    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-registry-certificates\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.735847 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735702    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-49fzr\" (UniqueName: \"kubernetes.io/projected/05906a72-1fb8-49fc-86c9-01d5fa92a7cd-kube-api-access-49fzr\") pod \"ingress-canary-bs42h\" (UID: \"05906a72-1fb8-49fc-86c9-01d5fa92a7cd\") " pod="openshift-ingress-canary/ingress-canary-bs42h"
Apr 24 22:30:42.735847 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735719    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-data-volume\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.735847 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735738    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-config-volume\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.735847 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735789    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fjzmh\" (UniqueName: \"kubernetes.io/projected/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-kube-api-access-fjzmh\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.735847 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735825    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-metrics-tls\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.736097 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735861    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-installation-pull-secrets\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.736097 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735898    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-ca-trust-extracted\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.736097 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735931    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-image-registry-private-configuration\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.736097 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735965    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-trusted-ca\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.736097 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.735989    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05906a72-1fb8-49fc-86c9-01d5fa92a7cd-cert\") pod \"ingress-canary-bs42h\" (UID: \"05906a72-1fb8-49fc-86c9-01d5fa92a7cd\") " pod="openshift-ingress-canary/ingress-canary-bs42h"
Apr 24 22:30:42.736097 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736035    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-crio-socket\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.736097 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736078    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-data-volume\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.736429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736109    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.736429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736158    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.736429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736185    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnplx\" (UniqueName: \"kubernetes.io/projected/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-kube-api-access-pnplx\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.736429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736212    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5lm5\" (UniqueName: \"kubernetes.io/projected/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-kube-api-access-p5lm5\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.736429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736272    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-registry-tls\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.736429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736314    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-bound-sa-token\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.736429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736314    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-ca-trust-extracted\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.736429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736366    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-tmp-dir\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.736794 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736678    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-registry-certificates\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.736971 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.736939    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.737029 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.737018    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-crio-socket\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.737144 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.737122    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-trusted-ca\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.740653 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.740627    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-image-registry-private-configuration\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.740732 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.740635    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.740770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.740753    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-installation-pull-secrets\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.740770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.740761    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-registry-tls\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.740835 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.740819    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05906a72-1fb8-49fc-86c9-01d5fa92a7cd-cert\") pod \"ingress-canary-bs42h\" (UID: \"05906a72-1fb8-49fc-86c9-01d5fa92a7cd\") " pod="openshift-ingress-canary/ingress-canary-bs42h"
Apr 24 22:30:42.770360 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.770282    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-bound-sa-token\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.770462 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.770391    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49fzr\" (UniqueName: \"kubernetes.io/projected/05906a72-1fb8-49fc-86c9-01d5fa92a7cd-kube-api-access-49fzr\") pod \"ingress-canary-bs42h\" (UID: \"05906a72-1fb8-49fc-86c9-01d5fa92a7cd\") " pod="openshift-ingress-canary/ingress-canary-bs42h"
Apr 24 22:30:42.770617 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.770595    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnplx\" (UniqueName: \"kubernetes.io/projected/19a6cd90-b20b-46e5-b7b5-f2e406db86b5-kube-api-access-pnplx\") pod \"insights-runtime-extractor-lp9hc\" (UID: \"19a6cd90-b20b-46e5-b7b5-f2e406db86b5\") " pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.772232 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.772214    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjzmh\" (UniqueName: \"kubernetes.io/projected/11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f-kube-api-access-fjzmh\") pod \"image-registry-879bc8cd9-gm7gg\" (UID: \"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f\") " pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.819880 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.819855    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:42.837640 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.837616    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-tmp-dir\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.837744 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.837653    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-config-volume\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.837744 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.837674    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-metrics-tls\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.837744 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.837733    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p5lm5\" (UniqueName: \"kubernetes.io/projected/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-kube-api-access-p5lm5\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.837991 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.837974    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-tmp-dir\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.838246 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.838224    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-config-volume\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.840523 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.840503    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-metrics-tls\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.848442 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.848421    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lp9hc"
Apr 24 22:30:42.854278 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.854253    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5lm5\" (UniqueName: \"kubernetes.io/projected/9639aef1-f7ec-48b0-90ee-98dfcfd3884a-kube-api-access-p5lm5\") pod \"dns-default-9gfb7\" (UID: \"9639aef1-f7ec-48b0-90ee-98dfcfd3884a\") " pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:42.868170 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.868134    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bs42h"
Apr 24 22:30:42.930301 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:42.929938    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:43.064789 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.064764    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-879bc8cd9-gm7gg"]
Apr 24 22:30:43.067895 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:30:43.067862    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ba3c59_40dc_4673_ac40_6d4b1ff9bb3f.slice/crio-b80bdc77b1d8fc8810dcf48ae4738e104a82ce0eda4efcb77e3679510ad4080e WatchSource:0}: Error finding container b80bdc77b1d8fc8810dcf48ae4738e104a82ce0eda4efcb77e3679510ad4080e: Status 404 returned error can't find the container with id b80bdc77b1d8fc8810dcf48ae4738e104a82ce0eda4efcb77e3679510ad4080e
Apr 24 22:30:43.081220 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.081200    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lp9hc"]
Apr 24 22:30:43.082170 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.082148    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bs42h"]
Apr 24 22:30:43.086547 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:30:43.086526    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05906a72_1fb8_49fc_86c9_01d5fa92a7cd.slice/crio-90be1186b732a607d8d616db236f73e6dbb5ed3fd5e96da790d450aaad6a6823 WatchSource:0}: Error finding container 90be1186b732a607d8d616db236f73e6dbb5ed3fd5e96da790d450aaad6a6823: Status 404 returned error can't find the container with id 90be1186b732a607d8d616db236f73e6dbb5ed3fd5e96da790d450aaad6a6823
Apr 24 22:30:43.095745 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:30:43.095726    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a6cd90_b20b_46e5_b7b5_f2e406db86b5.slice/crio-fde1a31b0abd3241856238d34a3fa92beb24e843e7813f8f2d645518e80b2eee WatchSource:0}: Error finding container fde1a31b0abd3241856238d34a3fa92beb24e843e7813f8f2d645518e80b2eee: Status 404 returned error can't find the container with id fde1a31b0abd3241856238d34a3fa92beb24e843e7813f8f2d645518e80b2eee
Apr 24 22:30:43.124384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.124364    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9gfb7"]
Apr 24 22:30:43.129395 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:30:43.129371    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9639aef1_f7ec_48b0_90ee_98dfcfd3884a.slice/crio-09a0dde574f6cc25d474b5b10f566f810a49a3d351265c076167334242d3fcc0 WatchSource:0}: Error finding container 09a0dde574f6cc25d474b5b10f566f810a49a3d351265c076167334242d3fcc0: Status 404 returned error can't find the container with id 09a0dde574f6cc25d474b5b10f566f810a49a3d351265c076167334242d3fcc0
Apr 24 22:30:43.228272 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.228240    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9gfb7" event={"ID":"9639aef1-f7ec-48b0-90ee-98dfcfd3884a","Type":"ContainerStarted","Data":"09a0dde574f6cc25d474b5b10f566f810a49a3d351265c076167334242d3fcc0"}
Apr 24 22:30:43.229284 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.229261    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bs42h" event={"ID":"05906a72-1fb8-49fc-86c9-01d5fa92a7cd","Type":"ContainerStarted","Data":"90be1186b732a607d8d616db236f73e6dbb5ed3fd5e96da790d450aaad6a6823"}
Apr 24 22:30:43.230522 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.230492    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lp9hc" event={"ID":"19a6cd90-b20b-46e5-b7b5-f2e406db86b5","Type":"ContainerStarted","Data":"1cc085f5a1cdd29b2523109ee63151678ee16f1ebc867d60b0cf587a495bda3c"}
Apr 24 22:30:43.230612 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.230528    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lp9hc" event={"ID":"19a6cd90-b20b-46e5-b7b5-f2e406db86b5","Type":"ContainerStarted","Data":"fde1a31b0abd3241856238d34a3fa92beb24e843e7813f8f2d645518e80b2eee"}
Apr 24 22:30:43.231594 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.231575    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg" event={"ID":"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f","Type":"ContainerStarted","Data":"4df60bff782cf96126ec0cbafb24fc4eeddc4643070f1696cfc73b4e84d8780d"}
Apr 24 22:30:43.231680 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.231602    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg" event={"ID":"11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f","Type":"ContainerStarted","Data":"b80bdc77b1d8fc8810dcf48ae4738e104a82ce0eda4efcb77e3679510ad4080e"}
Apr 24 22:30:43.231728 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.231699    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:30:43.273988 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.273910    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg" podStartSLOduration=1.273894287 podStartE2EDuration="1.273894287s" podCreationTimestamp="2026-04-24 22:30:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:30:43.27319697 +0000 UTC m=+46.699594503" watchObservedRunningTime="2026-04-24 22:30:43.273894287 +0000 UTC m=+46.700291819"
Apr 24 22:30:43.665048 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.665017    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"]
Apr 24 22:30:43.680127 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.680098    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"
Apr 24 22:30:43.685264 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.685238    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Apr 24 22:30:43.685594 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.685572    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-98mfv\""
Apr 24 22:30:43.685738 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.685705    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"]
Apr 24 22:30:43.746773 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.746740    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7129347e-47f7-4b88-8fb1-7fbba551e7b3-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-h9j56\" (UID: \"7129347e-47f7-4b88-8fb1-7fbba551e7b3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"
Apr 24 22:30:43.847294 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:43.847174    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7129347e-47f7-4b88-8fb1-7fbba551e7b3-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-h9j56\" (UID: \"7129347e-47f7-4b88-8fb1-7fbba551e7b3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"
Apr 24 22:30:43.847454 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:43.847332    2543 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Apr 24 22:30:43.847454 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:30:43.847424    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7129347e-47f7-4b88-8fb1-7fbba551e7b3-tls-certificates podName:7129347e-47f7-4b88-8fb1-7fbba551e7b3 nodeName:}" failed. No retries permitted until 2026-04-24 22:30:44.347408264 +0000 UTC m=+47.773805777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7129347e-47f7-4b88-8fb1-7fbba551e7b3-tls-certificates") pod "prometheus-operator-admission-webhook-57cf98b594-h9j56" (UID: "7129347e-47f7-4b88-8fb1-7fbba551e7b3") : secret "prometheus-operator-admission-webhook-tls" not found
Apr 24 22:30:44.038250 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.038223    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6bcc868b7-q5ch2"]
Apr 24 22:30:44.058291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.058268    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:30:44.058699 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.058679    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6bcc868b7-q5ch2"
Apr 24 22:30:44.058817 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.058723    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:30:44.059217 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.059198    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:30:44.068002 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.066931    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2wb7b\""
Apr 24 22:30:44.068002 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.066968    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 24 22:30:44.068002 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.067193    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Apr 24 22:30:44.068002 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.067449    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Apr 24 22:30:44.068002 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.067520    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-s5t4s\""
Apr 24 22:30:44.068002 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.067651    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 24 22:30:44.071155 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.071136    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-94q8c\""
Apr 24 22:30:44.071357 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.071317    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 24 22:30:44.074404 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.074385    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6bcc868b7-q5ch2"]
Apr 24 22:30:44.075120 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.075104    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Apr 24 22:30:44.148968 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.148938    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8rb\" (UniqueName: \"kubernetes.io/projected/034c86c9-fa44-48c0-b7aa-b0f441d9785e-kube-api-access-xp8rb\") pod \"downloads-6bcc868b7-q5ch2\" (UID: \"034c86c9-fa44-48c0-b7aa-b0f441d9785e\") " pod="openshift-console/downloads-6bcc868b7-q5ch2"
Apr 24 22:30:44.250258 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.250228    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8rb\" (UniqueName: \"kubernetes.io/projected/034c86c9-fa44-48c0-b7aa-b0f441d9785e-kube-api-access-xp8rb\") pod \"downloads-6bcc868b7-q5ch2\" (UID: \"034c86c9-fa44-48c0-b7aa-b0f441d9785e\") " pod="openshift-console/downloads-6bcc868b7-q5ch2"
Apr 24 22:30:44.264229 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.264201    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8rb\" (UniqueName: \"kubernetes.io/projected/034c86c9-fa44-48c0-b7aa-b0f441d9785e-kube-api-access-xp8rb\") pod \"downloads-6bcc868b7-q5ch2\" (UID: \"034c86c9-fa44-48c0-b7aa-b0f441d9785e\") " pod="openshift-console/downloads-6bcc868b7-q5ch2"
Apr 24 22:30:44.351702 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.351635    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7129347e-47f7-4b88-8fb1-7fbba551e7b3-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-h9j56\" (UID: \"7129347e-47f7-4b88-8fb1-7fbba551e7b3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"
Apr 24 22:30:44.353914 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.353895    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7129347e-47f7-4b88-8fb1-7fbba551e7b3-tls-certificates\") pod \"prometheus-operator-admission-webhook-57cf98b594-h9j56\" (UID: \"7129347e-47f7-4b88-8fb1-7fbba551e7b3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"
Apr 24 22:30:44.378936 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.378910    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6bcc868b7-q5ch2"
Apr 24 22:30:44.592099 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:44.592069    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"
Apr 24 22:30:45.908663 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:45.908638    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6bcc868b7-q5ch2"]
Apr 24 22:30:45.922479 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:45.922456    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"]
Apr 24 22:30:45.927451 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:30:45.927421    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7129347e_47f7_4b88_8fb1_7fbba551e7b3.slice/crio-8f0d673ebef360ffb38e6992ce29db20889db8081f17d470946d026b0975e8cf WatchSource:0}: Error finding container 8f0d673ebef360ffb38e6992ce29db20889db8081f17d470946d026b0975e8cf: Status 404 returned error can't find the container with id 8f0d673ebef360ffb38e6992ce29db20889db8081f17d470946d026b0975e8cf
Apr 24 22:30:46.241640 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:46.241563    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6bcc868b7-q5ch2" event={"ID":"034c86c9-fa44-48c0-b7aa-b0f441d9785e","Type":"ContainerStarted","Data":"8d0289093daf3c88cc62acd56f96ac73122d3d8f8172658c60ea2baf3d997281"}
Apr 24 22:30:46.243472 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:46.243443    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9gfb7" event={"ID":"9639aef1-f7ec-48b0-90ee-98dfcfd3884a","Type":"ContainerStarted","Data":"e25e100309d057b997dbabe98ee63c89f8d2d46d620afcfc0514904e66eaae14"}
Apr 24 22:30:46.243603 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:46.243477    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9gfb7" event={"ID":"9639aef1-f7ec-48b0-90ee-98dfcfd3884a","Type":"ContainerStarted","Data":"1db55dba6fd198bfbd5900dd1b9c65f60793f112ef47caa585990ded524e4a3d"}
Apr 24 22:30:46.243603 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:46.243586    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:46.244912 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:46.244884    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bs42h" event={"ID":"05906a72-1fb8-49fc-86c9-01d5fa92a7cd","Type":"ContainerStarted","Data":"d16b4eafab77485ffcf5b8c84204329fb386694ef4e224afe9fc3cea6b8aff28"}
Apr 24 22:30:46.246551 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:46.246532    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lp9hc" event={"ID":"19a6cd90-b20b-46e5-b7b5-f2e406db86b5","Type":"ContainerStarted","Data":"878b8192aad05a4a62857ba94170bcadd7804e48c2efe70a0f2f9d779cb8a8eb"}
Apr 24 22:30:46.247616 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:46.247593    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56" event={"ID":"7129347e-47f7-4b88-8fb1-7fbba551e7b3","Type":"ContainerStarted","Data":"8f0d673ebef360ffb38e6992ce29db20889db8081f17d470946d026b0975e8cf"}
Apr 24 22:30:46.281519 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:46.281457    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-9gfb7" podStartSLOduration=1.6595927449999999 podStartE2EDuration="4.281442973s" podCreationTimestamp="2026-04-24 22:30:42 +0000 UTC" firstStartedPulling="2026-04-24 22:30:43.131022905 +0000 UTC m=+46.557420413" lastFinishedPulling="2026-04-24 22:30:45.752873133 +0000 UTC m=+49.179270641" observedRunningTime="2026-04-24 22:30:46.279788648 +0000 UTC m=+49.706186175" watchObservedRunningTime="2026-04-24 22:30:46.281442973 +0000 UTC m=+49.707840505"
Apr 24 22:30:46.306026 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:46.305975    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bs42h" podStartSLOduration=1.6368701049999999 podStartE2EDuration="4.305959194s" podCreationTimestamp="2026-04-24 22:30:42 +0000 UTC" firstStartedPulling="2026-04-24 22:30:43.088232419 +0000 UTC m=+46.514629926" lastFinishedPulling="2026-04-24 22:30:45.7573215 +0000 UTC m=+49.183719015" observedRunningTime="2026-04-24 22:30:46.305751849 +0000 UTC m=+49.732149543" watchObservedRunningTime="2026-04-24 22:30:46.305959194 +0000 UTC m=+49.732356723"
Apr 24 22:30:48.255452 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:48.255413    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lp9hc" event={"ID":"19a6cd90-b20b-46e5-b7b5-f2e406db86b5","Type":"ContainerStarted","Data":"1e9cc2a0dcde1da72a7d86ba6bf171bb47b201ca4f39311ca8e5ea8137e15737"}
Apr 24 22:30:48.256872 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:48.256839    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56" event={"ID":"7129347e-47f7-4b88-8fb1-7fbba551e7b3","Type":"ContainerStarted","Data":"3ef33a672ad0f9e30848f26c9803ffc831e383bac076ff9cb8a87583c202bf87"}
Apr 24 22:30:48.257084 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:48.257064    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"
Apr 24 22:30:48.262928 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:48.262910    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56"
Apr 24 22:30:48.279901 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:48.279854    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lp9hc" podStartSLOduration=1.398740516 podStartE2EDuration="6.279841246s" podCreationTimestamp="2026-04-24 22:30:42 +0000 UTC" firstStartedPulling="2026-04-24 22:30:43.207218097 +0000 UTC m=+46.633615605" lastFinishedPulling="2026-04-24 22:30:48.088318827 +0000 UTC m=+51.514716335" observedRunningTime="2026-04-24 22:30:48.27838116 +0000 UTC m=+51.704778690" watchObservedRunningTime="2026-04-24 22:30:48.279841246 +0000 UTC m=+51.706238776"
Apr 24 22:30:53.831291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.831221    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-57cf98b594-h9j56" podStartSLOduration=8.669886232 podStartE2EDuration="10.831195666s" podCreationTimestamp="2026-04-24 22:30:43 +0000 UTC" firstStartedPulling="2026-04-24 22:30:45.929441689 +0000 UTC m=+49.355839197" lastFinishedPulling="2026-04-24 22:30:48.090751122 +0000 UTC m=+51.517148631" observedRunningTime="2026-04-24 22:30:48.304508813 +0000 UTC m=+51.730906344" watchObservedRunningTime="2026-04-24 22:30:53.831195666 +0000 UTC m=+57.257593197"
Apr 24 22:30:53.831761 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.831569    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qth7q"]
Apr 24 22:30:53.870656 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.870632    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:53.878606 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.878583    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Apr 24 22:30:53.881148 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.881128    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Apr 24 22:30:53.881253 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.881154    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Apr 24 22:30:53.881253 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.881214    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Apr 24 22:30:53.881253 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.881246    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Apr 24 22:30:53.881430 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.881154    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fcsn5\""
Apr 24 22:30:53.882514 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.882359    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Apr 24 22:30:53.933252 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.933125    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-accelerators-collector-config\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:53.933252 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.933182    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762ece34-73cf-4ee7-b126-0c07a51efe69-sys\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:53.933454 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.933254    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-textfile\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:53.933454 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.933365    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/762ece34-73cf-4ee7-b126-0c07a51efe69-metrics-client-ca\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:53.933454 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.933407    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/762ece34-73cf-4ee7-b126-0c07a51efe69-root\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:53.933597 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.933443    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-wtmp\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:53.933597 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.933533    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:53.933597 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.933582    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-tls\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:53.933721 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:53.933611    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh4k8\" (UniqueName: \"kubernetes.io/projected/762ece34-73cf-4ee7-b126-0c07a51efe69-kube-api-access-gh4k8\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034228 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034193    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762ece34-73cf-4ee7-b126-0c07a51efe69-sys\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034414 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034246    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-textfile\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034414 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034300    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/762ece34-73cf-4ee7-b126-0c07a51efe69-metrics-client-ca\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034414 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034309    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762ece34-73cf-4ee7-b126-0c07a51efe69-sys\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034414 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034325    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/762ece34-73cf-4ee7-b126-0c07a51efe69-root\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034414 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034379    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/762ece34-73cf-4ee7-b126-0c07a51efe69-root\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034414 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034378    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-wtmp\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034414 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034412    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034443    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-tls\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034471    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gh4k8\" (UniqueName: \"kubernetes.io/projected/762ece34-73cf-4ee7-b126-0c07a51efe69-kube-api-access-gh4k8\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034490    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-wtmp\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034509    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-accelerators-collector-config\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034679    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-textfile\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.034974 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.034953    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/762ece34-73cf-4ee7-b126-0c07a51efe69-metrics-client-ca\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.035067 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.035051    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-accelerators-collector-config\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.037441 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.037416    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.037567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.037537    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/762ece34-73cf-4ee7-b126-0c07a51efe69-node-exporter-tls\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.110677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.110619    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh4k8\" (UniqueName: \"kubernetes.io/projected/762ece34-73cf-4ee7-b126-0c07a51efe69-kube-api-access-gh4k8\") pod \"node-exporter-qth7q\" (UID: \"762ece34-73cf-4ee7-b126-0c07a51efe69\") " pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.180974 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.180950    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qth7q"
Apr 24 22:30:54.272635 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:54.272604    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qth7q" event={"ID":"762ece34-73cf-4ee7-b126-0c07a51efe69","Type":"ContainerStarted","Data":"1ca5be3f3eb56dd461f8cf9a52131a41651d518a1b7605d6e4e96daf45fa03f0"}
Apr 24 22:30:56.202883 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.202850    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t72f2"
Apr 24 22:30:56.253113 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.253084    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9gfb7"
Apr 24 22:30:56.788780 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.788744    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"]
Apr 24 22:30:56.794071 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.794047    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.796945 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.796918    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Apr 24 22:30:56.797109 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.797085    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-glttm\""
Apr 24 22:30:56.797182 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.797122    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Apr 24 22:30:56.797234 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.797191    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5lb257da3biek\""
Apr 24 22:30:56.797397 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.797371    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Apr 24 22:30:56.797735 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.797716    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Apr 24 22:30:56.797932 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.797913    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Apr 24 22:30:56.829604 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.829579    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"]
Apr 24 22:30:56.857618 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.857593    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.857729 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.857674    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.857729 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.857712    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqjmm\" (UniqueName: \"kubernetes.io/projected/4767d757-18ad-4679-9809-9eb8d54fd794-kube-api-access-rqjmm\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.857839 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.857794    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.857839 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.857822    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4767d757-18ad-4679-9809-9eb8d54fd794-metrics-client-ca\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.857941 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.857863    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.857941 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.857888    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-grpc-tls\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.857941 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.857918    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-tls\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.959165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.959124    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.959165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.959156    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4767d757-18ad-4679-9809-9eb8d54fd794-metrics-client-ca\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.959420 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.959175    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.959420 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.959190    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-grpc-tls\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.959420 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.959209    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-tls\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.959420 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.959225    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.959420 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.959265    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.959420 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.959284    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rqjmm\" (UniqueName: \"kubernetes.io/projected/4767d757-18ad-4679-9809-9eb8d54fd794-kube-api-access-rqjmm\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.960313 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.960165    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4767d757-18ad-4679-9809-9eb8d54fd794-metrics-client-ca\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.963222 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.963202    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Apr 24 22:30:56.963775 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.963580    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Apr 24 22:30:56.963775 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.963580    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Apr 24 22:30:56.963775 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.963656    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Apr 24 22:30:56.963775 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.963585    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Apr 24 22:30:56.965365 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.965320    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5lb257da3biek\""
Apr 24 22:30:56.972959 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.972936    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqjmm\" (UniqueName: \"kubernetes.io/projected/4767d757-18ad-4679-9809-9eb8d54fd794-kube-api-access-rqjmm\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.973112 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.973085    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.974369 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.974326    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-grpc-tls\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.974462 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.974394    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-tls\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.975055 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.975034    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.975150 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.975068    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:56.975883 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:56.975861    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4767d757-18ad-4679-9809-9eb8d54fd794-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d5d8d8c58-rzj8p\" (UID: \"4767d757-18ad-4679-9809-9eb8d54fd794\") " pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:57.110404 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:57.110321    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-glttm\""
Apr 24 22:30:57.117871 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:57.117841    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:30:58.196292 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.196264    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"]
Apr 24 22:30:58.200687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.200662    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"
Apr 24 22:30:58.203027 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.203005    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Apr 24 22:30:58.203160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.203120    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-mvgbj\""
Apr 24 22:30:58.206785 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.206765    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"]
Apr 24 22:30:58.270139 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.270109    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d939fa26-4fe1-4af1-913c-cd1f44f0d81d-monitoring-plugin-cert\") pod \"monitoring-plugin-7dccd58f55-w2qcz\" (UID: \"d939fa26-4fe1-4af1-913c-cd1f44f0d81d\") " pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"
Apr 24 22:30:58.370824 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.370784    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d939fa26-4fe1-4af1-913c-cd1f44f0d81d-monitoring-plugin-cert\") pod \"monitoring-plugin-7dccd58f55-w2qcz\" (UID: \"d939fa26-4fe1-4af1-913c-cd1f44f0d81d\") " pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"
Apr 24 22:30:58.373643 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.373614    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d939fa26-4fe1-4af1-913c-cd1f44f0d81d-monitoring-plugin-cert\") pod \"monitoring-plugin-7dccd58f55-w2qcz\" (UID: \"d939fa26-4fe1-4af1-913c-cd1f44f0d81d\") " pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"
Apr 24 22:30:58.501668 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.501611    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9d6868c76-bqsqz"]
Apr 24 22:30:58.505100 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.505073    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.508981 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.508957    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Apr 24 22:30:58.509107 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.508996    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9q965njkif7p4\""
Apr 24 22:30:58.509262 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.509236    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Apr 24 22:30:58.509262 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.509253    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rmchs\""
Apr 24 22:30:58.509414 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.509292    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Apr 24 22:30:58.509528 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.509513    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Apr 24 22:30:58.510864 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.510843    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"
Apr 24 22:30:58.514979 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.514959    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9d6868c76-bqsqz"]
Apr 24 22:30:58.572496 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.572464    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-secret-metrics-server-client-certs\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.572635 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.572506    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-audit-log\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.572635 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.572536    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clvcr\" (UniqueName: \"kubernetes.io/projected/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-kube-api-access-clvcr\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.572635 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.572579    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.572752 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.572642    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-secret-metrics-server-tls\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.572752 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.572706    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-metrics-server-audit-profiles\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.572752 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.572730    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-client-ca-bundle\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.673938 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.673898    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-metrics-server-audit-profiles\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.674112 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.673946    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-client-ca-bundle\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.674112 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.674014    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-secret-metrics-server-client-certs\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.674112 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.674041    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-audit-log\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.674112 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.674065    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-clvcr\" (UniqueName: \"kubernetes.io/projected/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-kube-api-access-clvcr\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.674112 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.674105    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.674394 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.674129    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-secret-metrics-server-tls\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.674842 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.674820    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-audit-log\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.675035 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.675015    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-metrics-server-audit-profiles\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.675742 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.675706    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.677742 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.677719    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-secret-metrics-server-tls\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.677835 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.677763    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-client-ca-bundle\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.678968 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.678943    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-secret-metrics-server-client-certs\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.684206 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.684185    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvcr\" (UniqueName: \"kubernetes.io/projected/3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b-kube-api-access-clvcr\") pod \"metrics-server-9d6868c76-bqsqz\" (UID: \"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b\") " pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:58.816807 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:58.816728    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:30:59.993297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:30:59.993264    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 22:31:00.027452 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.027422    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 22:31:00.027611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.027601    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.030244 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.030218    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Apr 24 22:31:00.030461 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.030440    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Apr 24 22:31:00.030545 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.030528    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Apr 24 22:31:00.030594 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.030532    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7p2dt\""
Apr 24 22:31:00.030849 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.030832    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Apr 24 22:31:00.031101 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.031084    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Apr 24 22:31:00.031740 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.031719    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Apr 24 22:31:00.032331 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.032314    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Apr 24 22:31:00.033984 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.033353    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Apr 24 22:31:00.033984 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.033488    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Apr 24 22:31:00.036090 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.035512    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fpft97vognomt\""
Apr 24 22:31:00.036090 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.035729    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Apr 24 22:31:00.037074 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.037054    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Apr 24 22:31:00.039457 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.039436    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Apr 24 22:31:00.186907 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.186867    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187092 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.186939    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187092 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187037    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187189 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187089    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187189 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187122    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-config-out\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187189 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187150    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187233    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187258    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187465 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187392    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187465 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187438    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187483    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-config\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187504    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187525    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187557    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187749 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187593    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-web-config\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187749 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187648    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187749 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187702    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-kube-api-access-48rq9\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.187749 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.187728    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.288493 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.288409    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.288493 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.288457    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.288735 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.288502    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-config\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.288735 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.288527    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289120 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289037    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289120 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289089    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289124    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-web-config\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289164    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289202    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-kube-api-access-48rq9\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289234    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289266    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289313    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289379    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289411    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289439    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-config-out\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289467    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289542    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.289589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.289571    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.290429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.290277    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.291011 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.290982    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.291667 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.291468    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.291996 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.291887    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.291996 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.291926    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.292434 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.292398    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.292675 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.292438    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.292675 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.292609    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.292955 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.292877    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.293281 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.293245    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.293897 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.293877    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-config\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.294253 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.294234    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-config-out\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.294763 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.294740    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.301457 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.301439    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-kube-api-access-48rq9\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.304790 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.304744    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-web-config\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.305033 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.305014    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.305161 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.305123    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.305260 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.305162    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:00.340319 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:00.340282    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:01.623847 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:01.623821    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 22:31:01.627297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:01.627273    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"]
Apr 24 22:31:01.762399 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:31:01.759446    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f12dbaf_f83b_4e9b_908f_98ee18698077.slice/crio-acda02808e4e896603179ba4c55711edf4e0b39433b94be23ced6a5160c0a66d WatchSource:0}: Error finding container acda02808e4e896603179ba4c55711edf4e0b39433b94be23ced6a5160c0a66d: Status 404 returned error can't find the container with id acda02808e4e896603179ba4c55711edf4e0b39433b94be23ced6a5160c0a66d
Apr 24 22:31:01.762399 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:31:01.759875    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd939fa26_4fe1_4af1_913c_cd1f44f0d81d.slice/crio-8222325b2e8d1f51c476a89c1dfda7e78bd1f53f8eb09bbff6c5a980fb9d0d61 WatchSource:0}: Error finding container 8222325b2e8d1f51c476a89c1dfda7e78bd1f53f8eb09bbff6c5a980fb9d0d61: Status 404 returned error can't find the container with id 8222325b2e8d1f51c476a89c1dfda7e78bd1f53f8eb09bbff6c5a980fb9d0d61
Apr 24 22:31:01.859362 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:01.859293    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9d6868c76-bqsqz"]
Apr 24 22:31:01.862194 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:31:01.862165    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9e8dd3_d002_4d9d_a0b9_3f3757b1fb5b.slice/crio-966f619e661fad0eeeb275d4d747ca0ab8f34f894a9e6982b39025e0073124a3 WatchSource:0}: Error finding container 966f619e661fad0eeeb275d4d747ca0ab8f34f894a9e6982b39025e0073124a3: Status 404 returned error can't find the container with id 966f619e661fad0eeeb275d4d747ca0ab8f34f894a9e6982b39025e0073124a3
Apr 24 22:31:01.867474 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:01.866434    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"]
Apr 24 22:31:01.877370 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:31:01.877317    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4767d757_18ad_4679_9809_9eb8d54fd794.slice/crio-02b457c8ca8258ad982c893493ed72bfd8010ecf443714c19221c214f4c7e070 WatchSource:0}: Error finding container 02b457c8ca8258ad982c893493ed72bfd8010ecf443714c19221c214f4c7e070: Status 404 returned error can't find the container with id 02b457c8ca8258ad982c893493ed72bfd8010ecf443714c19221c214f4c7e070
Apr 24 22:31:02.298703 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.298669    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6bcc868b7-q5ch2" event={"ID":"034c86c9-fa44-48c0-b7aa-b0f441d9785e","Type":"ContainerStarted","Data":"770e4d487b0ea066c2ad4d97d62cfbccb2b6d8f89e298654e77aeb5cde9e9d0b"}
Apr 24 22:31:02.299417 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.299372    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6bcc868b7-q5ch2"
Apr 24 22:31:02.300666 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.300641    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz" event={"ID":"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b","Type":"ContainerStarted","Data":"966f619e661fad0eeeb275d4d747ca0ab8f34f894a9e6982b39025e0073124a3"}
Apr 24 22:31:02.301704 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.301685    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz" event={"ID":"d939fa26-4fe1-4af1-913c-cd1f44f0d81d","Type":"ContainerStarted","Data":"8222325b2e8d1f51c476a89c1dfda7e78bd1f53f8eb09bbff6c5a980fb9d0d61"}
Apr 24 22:31:02.302812 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.302787    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerStarted","Data":"acda02808e4e896603179ba4c55711edf4e0b39433b94be23ced6a5160c0a66d"}
Apr 24 22:31:02.303663 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.303647    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p" event={"ID":"4767d757-18ad-4679-9809-9eb8d54fd794","Type":"ContainerStarted","Data":"02b457c8ca8258ad982c893493ed72bfd8010ecf443714c19221c214f4c7e070"}
Apr 24 22:31:02.316869 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.316830    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-6bcc868b7-q5ch2" podStartSLOduration=2.414308772 podStartE2EDuration="18.316817924s" podCreationTimestamp="2026-04-24 22:30:44 +0000 UTC" firstStartedPulling="2026-04-24 22:30:45.926326852 +0000 UTC m=+49.352724366" lastFinishedPulling="2026-04-24 22:31:01.828835998 +0000 UTC m=+65.255233518" observedRunningTime="2026-04-24 22:31:02.31627111 +0000 UTC m=+65.742668653" watchObservedRunningTime="2026-04-24 22:31:02.316817924 +0000 UTC m=+65.743215453"
Apr 24 22:31:02.318686 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.318666    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6bcc868b7-q5ch2"
Apr 24 22:31:02.912038 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.911964    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:31:02.914726 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.912072    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:31:02.915207 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.915184    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 24 22:31:02.916165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.916141    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 24 22:31:02.926241 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.926158    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 24 22:31:02.926712 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.926616    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df793786-616a-4001-9c32-2e4ebd101bb2-metrics-certs\") pod \"network-metrics-daemon-pxq7f\" (UID: \"df793786-616a-4001-9c32-2e4ebd101bb2\") " pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:31:02.939991 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.939967    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bxk\" (UniqueName: \"kubernetes.io/projected/e7d5adb7-ab2b-47df-859a-74f9de07e09b-kube-api-access-78bxk\") pod \"network-check-target-2lldp\" (UID: \"e7d5adb7-ab2b-47df-859a-74f9de07e09b\") " pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:31:02.973163 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.973126    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2wb7b\""
Apr 24 22:31:02.981740 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:02.981699    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxq7f"
Apr 24 22:31:03.005322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:03.005128    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-94q8c\""
Apr 24 22:31:03.013419 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:03.013401    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:31:03.309370 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:03.309100    2543 generic.go:358] "Generic (PLEG): container finished" podID="762ece34-73cf-4ee7-b126-0c07a51efe69" containerID="f2280feb529a163af449bb76cfa7053325eafd0e4274503814ccb9f0bb22b64b" exitCode=0
Apr 24 22:31:03.309764 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:03.309708    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qth7q" event={"ID":"762ece34-73cf-4ee7-b126-0c07a51efe69","Type":"ContainerDied","Data":"f2280feb529a163af449bb76cfa7053325eafd0e4274503814ccb9f0bb22b64b"}
Apr 24 22:31:04.239698 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:04.239037    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-879bc8cd9-gm7gg"
Apr 24 22:31:05.875191 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:05.875164    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2lldp"]
Apr 24 22:31:05.906785 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:05.906728    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pxq7f"]
Apr 24 22:31:06.011333 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:31:06.011292    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf793786_616a_4001_9c32_2e4ebd101bb2.slice/crio-13239228b2614206b01b0bbb01403cce291d2a854b3040e8bf889fe727aa8302 WatchSource:0}: Error finding container 13239228b2614206b01b0bbb01403cce291d2a854b3040e8bf889fe727aa8302: Status 404 returned error can't find the container with id 13239228b2614206b01b0bbb01403cce291d2a854b3040e8bf889fe727aa8302
Apr 24 22:31:06.011605 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:31:06.011578    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d5adb7_ab2b_47df_859a_74f9de07e09b.slice/crio-ddf6e3060e96017bc53962a0d215f364ddefe94d3b9e044742458a951cc2737b WatchSource:0}: Error finding container ddf6e3060e96017bc53962a0d215f364ddefe94d3b9e044742458a951cc2737b: Status 404 returned error can't find the container with id ddf6e3060e96017bc53962a0d215f364ddefe94d3b9e044742458a951cc2737b
Apr 24 22:31:06.324471 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.324435    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qth7q" event={"ID":"762ece34-73cf-4ee7-b126-0c07a51efe69","Type":"ContainerStarted","Data":"a561302564a814e6b42f2ad81eaf7a4f58e1286beee32e59b02adc8a08f77c98"}
Apr 24 22:31:06.324582 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.324475    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qth7q" event={"ID":"762ece34-73cf-4ee7-b126-0c07a51efe69","Type":"ContainerStarted","Data":"16fbca8276ee4c14b1fe2569a55d49fba6d855108b79ebcbca4a7c7d806346d1"}
Apr 24 22:31:06.327572 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.327548    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p" event={"ID":"4767d757-18ad-4679-9809-9eb8d54fd794","Type":"ContainerStarted","Data":"9cc59dc45c1982395b276ae4e2863d478bcf50ae4ef03815f94063e2cfdeb59d"}
Apr 24 22:31:06.327670 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.327580    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p" event={"ID":"4767d757-18ad-4679-9809-9eb8d54fd794","Type":"ContainerStarted","Data":"4c5943888aae87d3962fd3f87d2eeb9f840394218458370590dc58f3a6d2793d"}
Apr 24 22:31:06.328778 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.328755    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pxq7f" event={"ID":"df793786-616a-4001-9c32-2e4ebd101bb2","Type":"ContainerStarted","Data":"13239228b2614206b01b0bbb01403cce291d2a854b3040e8bf889fe727aa8302"}
Apr 24 22:31:06.330416 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.330390    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz" event={"ID":"3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b","Type":"ContainerStarted","Data":"a5d8d5b7e6bbd5d2224a0d1c1f0bae0484aa89fb34d43cfd74cadcf01746bab4"}
Apr 24 22:31:06.332007 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.331912    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz" event={"ID":"d939fa26-4fe1-4af1-913c-cd1f44f0d81d","Type":"ContainerStarted","Data":"df0b8d033dfec48a70d4259b4b7f1ce829d5fe4817997d7b9debb66a6534a4ab"}
Apr 24 22:31:06.332214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.332183    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"
Apr 24 22:31:06.339009 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.338982    2543 generic.go:358] "Generic (PLEG): container finished" podID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerID="1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63" exitCode=0
Apr 24 22:31:06.339098 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.339052    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerDied","Data":"1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63"}
Apr 24 22:31:06.339254 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.339228    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz"
Apr 24 22:31:06.340763 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.340735    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2lldp" event={"ID":"e7d5adb7-ab2b-47df-859a-74f9de07e09b","Type":"ContainerStarted","Data":"ddf6e3060e96017bc53962a0d215f364ddefe94d3b9e044742458a951cc2737b"}
Apr 24 22:31:06.366376 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.366233    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qth7q" podStartSLOduration=5.176801981 podStartE2EDuration="13.366217628s" podCreationTimestamp="2026-04-24 22:30:53 +0000 UTC" firstStartedPulling="2026-04-24 22:30:54.19355599 +0000 UTC m=+57.619953501" lastFinishedPulling="2026-04-24 22:31:02.382971634 +0000 UTC m=+65.809369148" observedRunningTime="2026-04-24 22:31:06.363939554 +0000 UTC m=+69.790337082" watchObservedRunningTime="2026-04-24 22:31:06.366217628 +0000 UTC m=+69.792615160"
Apr 24 22:31:06.433538 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.433442    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7dccd58f55-w2qcz" podStartSLOduration=4.518314895 podStartE2EDuration="8.43342458s" podCreationTimestamp="2026-04-24 22:30:58 +0000 UTC" firstStartedPulling="2026-04-24 22:31:01.778140119 +0000 UTC m=+65.204537630" lastFinishedPulling="2026-04-24 22:31:05.693249805 +0000 UTC m=+69.119647315" observedRunningTime="2026-04-24 22:31:06.431324557 +0000 UTC m=+69.857722179" watchObservedRunningTime="2026-04-24 22:31:06.43342458 +0000 UTC m=+69.859822111"
Apr 24 22:31:06.459214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:06.459161    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz" podStartSLOduration=4.629333261 podStartE2EDuration="8.459141196s" podCreationTimestamp="2026-04-24 22:30:58 +0000 UTC" firstStartedPulling="2026-04-24 22:31:01.864470074 +0000 UTC m=+65.290867585" lastFinishedPulling="2026-04-24 22:31:05.694278008 +0000 UTC m=+69.120675520" observedRunningTime="2026-04-24 22:31:06.458861613 +0000 UTC m=+69.885259144" watchObservedRunningTime="2026-04-24 22:31:06.459141196 +0000 UTC m=+69.885538726"
Apr 24 22:31:07.353223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:07.353172    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p" event={"ID":"4767d757-18ad-4679-9809-9eb8d54fd794","Type":"ContainerStarted","Data":"6427ed18940e281b6090d0601a09b0c3bddb06614b796f2ebd32c021f68c526c"}
Apr 24 22:31:08.360178 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:08.360124    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p" event={"ID":"4767d757-18ad-4679-9809-9eb8d54fd794","Type":"ContainerStarted","Data":"1845b44160fbcd739b1bb403605e7e7108cd4e47db19b11daefa5236fe83a419"}
Apr 24 22:31:09.372611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:09.372542    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p" event={"ID":"4767d757-18ad-4679-9809-9eb8d54fd794","Type":"ContainerStarted","Data":"f46a79e5b10b77dede9dbda079d9ecc55c4bff84a256e61456d2aa6ec35e54ab"}
Apr 24 22:31:09.376158 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:09.376129    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pxq7f" event={"ID":"df793786-616a-4001-9c32-2e4ebd101bb2","Type":"ContainerStarted","Data":"44168c2b7b2fcce175babe6074284b754d7563cdb81a79715b37d96a88cc87f9"}
Apr 24 22:31:12.388949 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.388922    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerStarted","Data":"cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744"}
Apr 24 22:31:12.389324 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.388959    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerStarted","Data":"6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488"}
Apr 24 22:31:12.389324 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.388969    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerStarted","Data":"244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1"}
Apr 24 22:31:12.389324 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.388981    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerStarted","Data":"e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92"}
Apr 24 22:31:12.389324 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.388993    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerStarted","Data":"e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d"}
Apr 24 22:31:12.390493 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.390468    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2lldp" event={"ID":"e7d5adb7-ab2b-47df-859a-74f9de07e09b","Type":"ContainerStarted","Data":"b7d688a9df686bdcddd9b54432944b8303d8d27f46ec15b794e271aa33fcaca5"}
Apr 24 22:31:12.390605 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.390535    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:31:12.393747 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.393689    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p" event={"ID":"4767d757-18ad-4679-9809-9eb8d54fd794","Type":"ContainerStarted","Data":"e047768ee269638e2bd6322ff5eef5f0aad24b2387e695d2eeacb91a7e77352c"}
Apr 24 22:31:12.393937 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.393912    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:31:12.396212 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.396172    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pxq7f" event={"ID":"df793786-616a-4001-9c32-2e4ebd101bb2","Type":"ContainerStarted","Data":"728c0f4fb4c8826d3778849a180bdae8282f90bac557385206f12527a854e8c0"}
Apr 24 22:31:12.400202 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.400187    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p"
Apr 24 22:31:12.416282 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.416243    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2lldp" podStartSLOduration=69.809267827 podStartE2EDuration="1m15.41623316s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:31:06.013603148 +0000 UTC m=+69.440000661" lastFinishedPulling="2026-04-24 22:31:11.620568471 +0000 UTC m=+75.046965994" observedRunningTime="2026-04-24 22:31:12.415724097 +0000 UTC m=+75.842121628" watchObservedRunningTime="2026-04-24 22:31:12.41623316 +0000 UTC m=+75.842630690"
Apr 24 22:31:12.561892 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.561834    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-d5d8d8c58-rzj8p" podStartSLOduration=10.260518683 podStartE2EDuration="16.561813704s" podCreationTimestamp="2026-04-24 22:30:56 +0000 UTC" firstStartedPulling="2026-04-24 22:31:01.879359185 +0000 UTC m=+65.305756696" lastFinishedPulling="2026-04-24 22:31:08.180654209 +0000 UTC m=+71.607051717" observedRunningTime="2026-04-24 22:31:12.47080224 +0000 UTC m=+75.897199771" watchObservedRunningTime="2026-04-24 22:31:12.561813704 +0000 UTC m=+75.988211235"
Apr 24 22:31:12.825102 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.825069    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:31:12.827823 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.827800    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Apr 24 22:31:12.838253 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.838231    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d17a86f7-36e1-4352-92cf-748b8c466f7e-original-pull-secret\") pod \"global-pull-secret-syncer-kcq9z\" (UID: \"d17a86f7-36e1-4352-92cf-748b8c466f7e\") " pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:31:12.887823 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:12.887798    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kcq9z"
Apr 24 22:31:13.014949 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:13.014902    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pxq7f" podStartSLOduration=73.851376635 podStartE2EDuration="1m16.014885549s" podCreationTimestamp="2026-04-24 22:29:57 +0000 UTC" firstStartedPulling="2026-04-24 22:31:06.013492604 +0000 UTC m=+69.439890112" lastFinishedPulling="2026-04-24 22:31:08.177001514 +0000 UTC m=+71.603399026" observedRunningTime="2026-04-24 22:31:12.563298612 +0000 UTC m=+75.989696143" watchObservedRunningTime="2026-04-24 22:31:13.014885549 +0000 UTC m=+76.441283084"
Apr 24 22:31:13.015824 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:13.015800    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kcq9z"]
Apr 24 22:31:13.020887 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:31:13.020858    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17a86f7_36e1_4352_92cf_748b8c466f7e.slice/crio-03424a6dbe9133e997fd1593a840c3294f7604b9bee68746fc66ecd327c7dd0b WatchSource:0}: Error finding container 03424a6dbe9133e997fd1593a840c3294f7604b9bee68746fc66ecd327c7dd0b: Status 404 returned error can't find the container with id 03424a6dbe9133e997fd1593a840c3294f7604b9bee68746fc66ecd327c7dd0b
Apr 24 22:31:13.400069 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:13.400035    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kcq9z" event={"ID":"d17a86f7-36e1-4352-92cf-748b8c466f7e","Type":"ContainerStarted","Data":"03424a6dbe9133e997fd1593a840c3294f7604b9bee68746fc66ecd327c7dd0b"}
Apr 24 22:31:13.403080 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:13.403042    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerStarted","Data":"bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e"}
Apr 24 22:31:13.460480 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:13.460438    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.611790923 podStartE2EDuration="14.460426845s" podCreationTimestamp="2026-04-24 22:30:59 +0000 UTC" firstStartedPulling="2026-04-24 22:31:01.778200165 +0000 UTC m=+65.204597680" lastFinishedPulling="2026-04-24 22:31:11.62683608 +0000 UTC m=+75.053233602" observedRunningTime="2026-04-24 22:31:13.453650942 +0000 UTC m=+76.880048473" watchObservedRunningTime="2026-04-24 22:31:13.460426845 +0000 UTC m=+76.886824375"
Apr 24 22:31:15.341249 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:15.341208    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:31:17.420872 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:17.420796    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kcq9z" event={"ID":"d17a86f7-36e1-4352-92cf-748b8c466f7e","Type":"ContainerStarted","Data":"c412187f120fe91e359f7c3f11e3da5f5b97f22c78d7a70974427f11917ebffe"}
Apr 24 22:31:17.437049 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:17.437009    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kcq9z" podStartSLOduration=65.303601121 podStartE2EDuration="1m9.43699545s" podCreationTimestamp="2026-04-24 22:30:08 +0000 UTC" firstStartedPulling="2026-04-24 22:31:13.022727249 +0000 UTC m=+76.449124756" lastFinishedPulling="2026-04-24 22:31:17.156121577 +0000 UTC m=+80.582519085" observedRunningTime="2026-04-24 22:31:17.436056621 +0000 UTC m=+80.862454152" watchObservedRunningTime="2026-04-24 22:31:17.43699545 +0000 UTC m=+80.863392980"
Apr 24 22:31:18.817555 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:18.817530    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:31:18.817555 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:18.817560    2543 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:31:38.822722 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:38.822692    2543 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:31:38.826656 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:38.826635    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9d6868c76-bqsqz"
Apr 24 22:31:43.406184 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:31:43.406155    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2lldp"
Apr 24 22:32:00.341063 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:00.341026    2543 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:00.359923 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:00.359902    2543 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:00.552768 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:00.552741    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:18.383537 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.383507    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 22:32:18.384021 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.383931    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="config-reloader" containerID="cri-o://e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92" gracePeriod=600
Apr 24 22:32:18.384021 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.383931    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy" containerID="cri-o://cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744" gracePeriod=600
Apr 24 22:32:18.384021 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.383971    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="thanos-sidecar" containerID="cri-o://244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1" gracePeriod=600
Apr 24 22:32:18.384172 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.384030    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy-web" containerID="cri-o://6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488" gracePeriod=600
Apr 24 22:32:18.384172 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.384005    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e" gracePeriod=600
Apr 24 22:32:18.384172 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.384010    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="prometheus" containerID="cri-o://e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d" gracePeriod=600
Apr 24 22:32:18.588160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.588130    2543 generic.go:358] "Generic (PLEG): container finished" podID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerID="6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488" exitCode=0
Apr 24 22:32:18.588160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.588153    2543 generic.go:358] "Generic (PLEG): container finished" podID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerID="244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1" exitCode=0
Apr 24 22:32:18.588160 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.588161    2543 generic.go:358] "Generic (PLEG): container finished" podID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerID="e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92" exitCode=0
Apr 24 22:32:18.588356 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.588203    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerDied","Data":"6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488"}
Apr 24 22:32:18.588356 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.588235    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerDied","Data":"244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1"}
Apr 24 22:32:18.588356 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:18.588244    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerDied","Data":"e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92"}
Apr 24 22:32:19.593258 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:19.593231    2543 generic.go:358] "Generic (PLEG): container finished" podID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerID="bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e" exitCode=0
Apr 24 22:32:19.593258 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:19.593253    2543 generic.go:358] "Generic (PLEG): container finished" podID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerID="cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744" exitCode=0
Apr 24 22:32:19.593567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:19.593273    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerDied","Data":"bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e"}
Apr 24 22:32:19.593567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:19.593294    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerDied","Data":"cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744"}
Apr 24 22:32:20.539005 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:20.538963    2543 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Apr 24 22:32:20.539950 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:20.539917    2543 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Apr 24 22:32:20.540757 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:20.540730    2543 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Apr 24 22:32:20.540860 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:20.540774    2543 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="prometheus" probeResult="unknown"
Apr 24 22:32:25.538760 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:25.538717    2543 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Apr 24 22:32:25.539627 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:25.539599    2543 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Apr 24 22:32:25.540446 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:25.540422    2543 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Apr 24 22:32:25.540522 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:25.540457    2543 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="prometheus" probeResult="unknown"
Apr 24 22:32:27.125203 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.125181    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.256332 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256309    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-grpc-tls\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256510 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256361    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-config-out\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256510 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256382    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-serving-certs-ca-bundle\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256510 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256410    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-metrics-client-ca\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256510 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256432    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-config\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256510 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256454    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-web-config\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256510 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256484    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-rulefiles-0\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256513    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256543    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256577    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-db\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256602    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-metrics-client-certs\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256628    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-kube-rbac-proxy\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256678    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-trusted-ca-bundle\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256733    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-thanos-prometheus-http-client-file\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256764    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-kube-api-access-48rq9\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256782    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256792    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-tls-assets\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.256815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256817    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-tls\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.257390 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256848    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-kubelet-serving-ca-bundle\") pod \"2f12dbaf-f83b-4e9b-908f-98ee18698077\" (UID: \"2f12dbaf-f83b-4e9b-908f-98ee18698077\") "
Apr 24 22:32:27.257390 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.256885    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:32:27.257390 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.257069    2543 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.257390 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.257206    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:32:27.257390 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.257380    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:32:27.260053 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.259225    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:32:27.260053 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.259285    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-config" (OuterVolumeSpecName: "config") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:32:27.260611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.260576    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:32:27.260944 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.260891    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:32:27.261356 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.261305    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:32:27.261734 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.261655    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-config-out" (OuterVolumeSpecName: "config-out") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:32:27.261885 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.261843    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:32:27.262033 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.261993    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:32:27.262142 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.262080    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:32:27.262142 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.262104    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:32:27.262272 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.262256    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:32:27.262371 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.262306    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-kube-api-access-48rq9" (OuterVolumeSpecName: "kube-api-access-48rq9") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "kube-api-access-48rq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:32:27.262907 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.262888    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:32:27.271288 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.271255    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-web-config" (OuterVolumeSpecName: "web-config") pod "2f12dbaf-f83b-4e9b-908f-98ee18698077" (UID: "2f12dbaf-f83b-4e9b-908f-98ee18698077"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:32:27.357453 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357435    2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357529 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357458    2543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357529 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357469    2543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357529 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357478    2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-k8s-db\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357529 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357487    2543 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-metrics-client-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357529 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357496    2543 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-kube-rbac-proxy\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357529 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357505    2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357529 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357515    2543 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357529 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357523    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-kube-api-access-48rq9\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357533    2543 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f12dbaf-f83b-4e9b-908f-98ee18698077-tls-assets\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357542    2543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357550    2543 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357558    2543 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-secret-grpc-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357566    2543 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f12dbaf-f83b-4e9b-908f-98ee18698077-config-out\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357574    2543 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f12dbaf-f83b-4e9b-908f-98ee18698077-configmap-metrics-client-ca\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357583    2543 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.357770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.357590    2543 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f12dbaf-f83b-4e9b-908f-98ee18698077-web-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:32:27.621565 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.621496    2543 generic.go:358] "Generic (PLEG): container finished" podID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d" exitCode=0
Apr 24 22:32:27.621677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.621567    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerDied","Data":"e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d"}
Apr 24 22:32:27.621677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.621588    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2f12dbaf-f83b-4e9b-908f-98ee18698077","Type":"ContainerDied","Data":"acda02808e4e896603179ba4c55711edf4e0b39433b94be23ced6a5160c0a66d"}
Apr 24 22:32:27.621677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.621602    2543 scope.go:117] "RemoveContainer" containerID="bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e"
Apr 24 22:32:27.621677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.621605    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.629848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.629829    2543 scope.go:117] "RemoveContainer" containerID="cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744"
Apr 24 22:32:27.636377 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.636359    2543 scope.go:117] "RemoveContainer" containerID="6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488"
Apr 24 22:32:27.642275 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.642257    2543 scope.go:117] "RemoveContainer" containerID="244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1"
Apr 24 22:32:27.644181 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.644153    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 22:32:27.648107 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.648086    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 22:32:27.649011 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.648996    2543 scope.go:117] "RemoveContainer" containerID="e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92"
Apr 24 22:32:27.655145 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.655126    2543 scope.go:117] "RemoveContainer" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d"
Apr 24 22:32:27.661328 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.661310    2543 scope.go:117] "RemoveContainer" containerID="1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63"
Apr 24 22:32:27.667165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.667149    2543 scope.go:117] "RemoveContainer" containerID="bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e"
Apr 24 22:32:27.667439 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:27.667414    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e\": container with ID starting with bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e not found: ID does not exist" containerID="bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e"
Apr 24 22:32:27.667531 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.667450    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e"} err="failed to get container status \"bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e\": rpc error: code = NotFound desc = could not find container \"bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e\": container with ID starting with bab0bca03b8e5227a177ae7f0dbccfa13972137c096ccfb4d4d82dfcfe65197e not found: ID does not exist"
Apr 24 22:32:27.667531 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.667493    2543 scope.go:117] "RemoveContainer" containerID="cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744"
Apr 24 22:32:27.667723 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:27.667704    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744\": container with ID starting with cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744 not found: ID does not exist" containerID="cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744"
Apr 24 22:32:27.667764 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.667728    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744"} err="failed to get container status \"cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744\": rpc error: code = NotFound desc = could not find container \"cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744\": container with ID starting with cc615ebb569927228e175794e1e36bb15dcd16774cd0648fff584db3b3c3b744 not found: ID does not exist"
Apr 24 22:32:27.667764 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.667743    2543 scope.go:117] "RemoveContainer" containerID="6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488"
Apr 24 22:32:27.667961 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:27.667947    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488\": container with ID starting with 6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488 not found: ID does not exist" containerID="6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488"
Apr 24 22:32:27.668010 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.667963    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488"} err="failed to get container status \"6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488\": rpc error: code = NotFound desc = could not find container \"6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488\": container with ID starting with 6fd9d8f5cfa7fc50981839e5254a749856272ce109e9e44b7fe0765a1802f488 not found: ID does not exist"
Apr 24 22:32:27.668010 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.667974    2543 scope.go:117] "RemoveContainer" containerID="244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1"
Apr 24 22:32:27.668196 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:27.668171    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1\": container with ID starting with 244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1 not found: ID does not exist" containerID="244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1"
Apr 24 22:32:27.668277 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.668207    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1"} err="failed to get container status \"244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1\": rpc error: code = NotFound desc = could not find container \"244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1\": container with ID starting with 244b6b04e692dcb715884e5aee09506fc4a685ad5e1ee5c4393f3145866f8df1 not found: ID does not exist"
Apr 24 22:32:27.668277 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.668229    2543 scope.go:117] "RemoveContainer" containerID="e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92"
Apr 24 22:32:27.668492 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:27.668476    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92\": container with ID starting with e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92 not found: ID does not exist" containerID="e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92"
Apr 24 22:32:27.668554 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.668500    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92"} err="failed to get container status \"e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92\": rpc error: code = NotFound desc = could not find container \"e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92\": container with ID starting with e49a52a077836818060a59434c93d62eab6da0b20db3e06be64cfa4084853d92 not found: ID does not exist"
Apr 24 22:32:27.668554 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.668520    2543 scope.go:117] "RemoveContainer" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d"
Apr 24 22:32:27.668755 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:27.668729    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d\": container with ID starting with e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d not found: ID does not exist" containerID="e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d"
Apr 24 22:32:27.668828 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.668754    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d"} err="failed to get container status \"e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d\": rpc error: code = NotFound desc = could not find container \"e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d\": container with ID starting with e2e4261fb4ef161a0fc3452c24456f509c528ca630275d7e055ea3296afd965d not found: ID does not exist"
Apr 24 22:32:27.668828 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.668771    2543 scope.go:117] "RemoveContainer" containerID="1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63"
Apr 24 22:32:27.669023 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:32:27.669000    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63\": container with ID starting with 1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63 not found: ID does not exist" containerID="1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63"
Apr 24 22:32:27.669119 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.669027    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63"} err="failed to get container status \"1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63\": rpc error: code = NotFound desc = could not find container \"1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63\": container with ID starting with 1943c06a4506e22ac267c102e676c31683a2888fa0eab7ab988f3df768e26d63 not found: ID does not exist"
Apr 24 22:32:27.671987 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.671968    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 22:32:27.672327 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672312    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy-web"
Apr 24 22:32:27.672397 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672331    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy-web"
Apr 24 22:32:27.672397 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672362    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="init-config-reloader"
Apr 24 22:32:27.672397 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672370    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="init-config-reloader"
Apr 24 22:32:27.672397 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672382    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy"
Apr 24 22:32:27.672397 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672390    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy"
Apr 24 22:32:27.672539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672400    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="thanos-sidecar"
Apr 24 22:32:27.672539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672408    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="thanos-sidecar"
Apr 24 22:32:27.672539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672448    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="config-reloader"
Apr 24 22:32:27.672539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672456    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="config-reloader"
Apr 24 22:32:27.672539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672469    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="prometheus"
Apr 24 22:32:27.672539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672477    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="prometheus"
Apr 24 22:32:27.672539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672496    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy-thanos"
Apr 24 22:32:27.672539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672503    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy-thanos"
Apr 24 22:32:27.672765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672559    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy-web"
Apr 24 22:32:27.672765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672570    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="thanos-sidecar"
Apr 24 22:32:27.672765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672582    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="config-reloader"
Apr 24 22:32:27.672765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672590    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="prometheus"
Apr 24 22:32:27.672765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672601    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy"
Apr 24 22:32:27.672765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.672609    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" containerName="kube-rbac-proxy-thanos"
Apr 24 22:32:27.678013 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.677999    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.680518 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.680501    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Apr 24 22:32:27.680851 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.680755    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Apr 24 22:32:27.680851 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.680803    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Apr 24 22:32:27.680851 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.680810    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Apr 24 22:32:27.681047 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.680858    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Apr 24 22:32:27.681047 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.680813    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Apr 24 22:32:27.681047 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.680973    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fpft97vognomt\""
Apr 24 22:32:27.681195 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.681182    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Apr 24 22:32:27.681289 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.681275    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Apr 24 22:32:27.681717 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.681682    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7p2dt\""
Apr 24 22:32:27.682572 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.682554    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Apr 24 22:32:27.685126 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.685108    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Apr 24 22:32:27.685262 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.685245    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Apr 24 22:32:27.687489 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.687469    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Apr 24 22:32:27.689762 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.689743    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 22:32:27.859968 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.859944    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860083 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.859973    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860083 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.859991    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860083 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860007    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860257 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860083    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-config\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860257 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860128    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860257 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860165    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/876ba290-f613-46a3-a6b5-a1071a85498f-config-out\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860257 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860190    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/876ba290-f613-46a3-a6b5-a1071a85498f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860257 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860216    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-web-config\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860257 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860242    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860268    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860292    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860320    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/876ba290-f613-46a3-a6b5-a1071a85498f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860397    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860431    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860479    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpsp\" (UniqueName: \"kubernetes.io/projected/876ba290-f613-46a3-a6b5-a1071a85498f-kube-api-access-qfpsp\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860512    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.860620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.860537    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.960981 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.960956    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-web-config\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961072 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.960985    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961072 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961003    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961072 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961028    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961251 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961232    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/876ba290-f613-46a3-a6b5-a1071a85498f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961325 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961284    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961325 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961319    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961489 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961376    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpsp\" (UniqueName: \"kubernetes.io/projected/876ba290-f613-46a3-a6b5-a1071a85498f-kube-api-access-qfpsp\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961614 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961595    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961725 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961709    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.961836 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961821    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.962018 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961947    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.962258 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.962242    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.962404 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.962387    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.962550 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.962533    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-config\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.962672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.962657    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.962768 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.962756    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/876ba290-f613-46a3-a6b5-a1071a85498f-config-out\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.962868 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.962854    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/876ba290-f613-46a3-a6b5-a1071a85498f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.963256 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.963237    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/876ba290-f613-46a3-a6b5-a1071a85498f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.964330 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.964304    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.964561 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.964539    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.964687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.964664    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.964764 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.962172    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.964824 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.964766    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.964985 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.964963    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.965053 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.961972    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.965053 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.962115    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.965246 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.965225    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.965575 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.965550    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/876ba290-f613-46a3-a6b5-a1071a85498f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.965731 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.965688    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-web-config\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.967291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.967271    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-config\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.967440 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.967419    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/876ba290-f613-46a3-a6b5-a1071a85498f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.967703 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.967686    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.967767 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.967691    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/876ba290-f613-46a3-a6b5-a1071a85498f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.967767 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.967727    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/876ba290-f613-46a3-a6b5-a1071a85498f-config-out\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.973300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.973278    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpsp\" (UniqueName: \"kubernetes.io/projected/876ba290-f613-46a3-a6b5-a1071a85498f-kube-api-access-qfpsp\") pod \"prometheus-k8s-0\" (UID: \"876ba290-f613-46a3-a6b5-a1071a85498f\") " pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:27.988194 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:27.988173    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:32:28.151290 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:28.151259    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Apr 24 22:32:28.154433 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:32:28.154404    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876ba290_f613_46a3_a6b5_a1071a85498f.slice/crio-a968fe14f24fa32bb0d4a4305dd38d93e7ad2dda090bb00cb5c517a551fa159b WatchSource:0}: Error finding container a968fe14f24fa32bb0d4a4305dd38d93e7ad2dda090bb00cb5c517a551fa159b: Status 404 returned error can't find the container with id a968fe14f24fa32bb0d4a4305dd38d93e7ad2dda090bb00cb5c517a551fa159b
Apr 24 22:32:28.625836 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:28.625804    2543 generic.go:358] "Generic (PLEG): container finished" podID="876ba290-f613-46a3-a6b5-a1071a85498f" containerID="cb33b132989e16f4f74f77ecf51e2db65022f43a25211d8cf06e10fe789e3abd" exitCode=0
Apr 24 22:32:28.625836 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:28.625838    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"876ba290-f613-46a3-a6b5-a1071a85498f","Type":"ContainerDied","Data":"cb33b132989e16f4f74f77ecf51e2db65022f43a25211d8cf06e10fe789e3abd"}
Apr 24 22:32:28.626052 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:28.625857    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"876ba290-f613-46a3-a6b5-a1071a85498f","Type":"ContainerStarted","Data":"a968fe14f24fa32bb0d4a4305dd38d93e7ad2dda090bb00cb5c517a551fa159b"}
Apr 24 22:32:29.059829 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:29.059789    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f12dbaf-f83b-4e9b-908f-98ee18698077" path="/var/lib/kubelet/pods/2f12dbaf-f83b-4e9b-908f-98ee18698077/volumes"
Apr 24 22:32:29.632095 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:29.632063    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"876ba290-f613-46a3-a6b5-a1071a85498f","Type":"ContainerStarted","Data":"f8f6dcd214814a6d48c5a4c76fd5fdc1ee04cea7396120855463ccafc9667a6f"}
Apr 24 22:32:29.632095 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:29.632096    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"876ba290-f613-46a3-a6b5-a1071a85498f","Type":"ContainerStarted","Data":"4ffe1ac1aa6a2e5423603a62069d839b9359cf34c3bf4a231f8d9bad1c87704d"}
Apr 24 22:32:29.632491 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:29.632105    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"876ba290-f613-46a3-a6b5-a1071a85498f","Type":"ContainerStarted","Data":"e4e4e92a415c94ee4dde4df9dee3d481f733dc44e1fd87335a751b125901c744"}
Apr 24 22:32:29.632491 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:29.632114    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"876ba290-f613-46a3-a6b5-a1071a85498f","Type":"ContainerStarted","Data":"4abb3a28621d5384b48bbb181dc59b58664663d71730d464a2868c1fd65ffd42"}
Apr 24 22:32:29.632491 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:29.632121    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"876ba290-f613-46a3-a6b5-a1071a85498f","Type":"ContainerStarted","Data":"dc49c2cbfb3406f28faa59e3fa5c03238a27cf98c3eb329fc802d035b84beedc"}
Apr 24 22:32:29.632491 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:29.632129    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"876ba290-f613-46a3-a6b5-a1071a85498f","Type":"ContainerStarted","Data":"910eab879b9c5e0e13e6391f590be90a6a209f2e4ed7d992ed825e4792b1462e"}
Apr 24 22:32:29.672219 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:29.672165    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.672147723 podStartE2EDuration="2.672147723s" podCreationTimestamp="2026-04-24 22:32:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:32:29.668450658 +0000 UTC m=+153.094848187" watchObservedRunningTime="2026-04-24 22:32:29.672147723 +0000 UTC m=+153.098545253"
Apr 24 22:32:32.989128 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:32:32.989097    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:33:27.988735 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:33:27.988700    2543 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:33:28.004128 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:33:28.004106    2543 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:33:28.805386 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:33:28.805359    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Apr 24 22:34:56.979392 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:34:56.979362    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:34:56.979913 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:34:56.979451    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:34:56.982551 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:34:56.982532    2543 kubelet.go:1628] "Image garbage collection succeeded"
Apr 24 22:36:01.328069 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.327989    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-86cc847c5c-88p77"]
Apr 24 22:36:01.331022 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.331007    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:36:01.334050 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.334032    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dqlzb\""
Apr 24 22:36:01.334395 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.334382    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Apr 24 22:36:01.335035 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.335016    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Apr 24 22:36:01.335103 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.335027    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Apr 24 22:36:01.347681 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.347661    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-86cc847c5c-88p77"]
Apr 24 22:36:01.380069 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.380050    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj9t9\" (UniqueName: \"kubernetes.io/projected/2acbce06-e4b4-43e9-ae93-74c97e292076-kube-api-access-kj9t9\") pod \"seaweedfs-86cc847c5c-88p77\" (UID: \"2acbce06-e4b4-43e9-ae93-74c97e292076\") " pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:36:01.380164 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.380078    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2acbce06-e4b4-43e9-ae93-74c97e292076-data\") pod \"seaweedfs-86cc847c5c-88p77\" (UID: \"2acbce06-e4b4-43e9-ae93-74c97e292076\") " pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:36:01.481430 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.481400    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kj9t9\" (UniqueName: \"kubernetes.io/projected/2acbce06-e4b4-43e9-ae93-74c97e292076-kube-api-access-kj9t9\") pod \"seaweedfs-86cc847c5c-88p77\" (UID: \"2acbce06-e4b4-43e9-ae93-74c97e292076\") " pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:36:01.481531 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.481439    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2acbce06-e4b4-43e9-ae93-74c97e292076-data\") pod \"seaweedfs-86cc847c5c-88p77\" (UID: \"2acbce06-e4b4-43e9-ae93-74c97e292076\") " pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:36:01.481767 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.481751    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2acbce06-e4b4-43e9-ae93-74c97e292076-data\") pod \"seaweedfs-86cc847c5c-88p77\" (UID: \"2acbce06-e4b4-43e9-ae93-74c97e292076\") " pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:36:01.490502 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.490478    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj9t9\" (UniqueName: \"kubernetes.io/projected/2acbce06-e4b4-43e9-ae93-74c97e292076-kube-api-access-kj9t9\") pod \"seaweedfs-86cc847c5c-88p77\" (UID: \"2acbce06-e4b4-43e9-ae93-74c97e292076\") " pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:36:01.639780 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.639725    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:36:01.771843 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.771816    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-86cc847c5c-88p77"]
Apr 24 22:36:01.775319 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:36:01.775293    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acbce06_e4b4_43e9_ae93_74c97e292076.slice/crio-132c6840955d50a0cfa4916f87c0d3a8faedeb20395873ab3d62e9f2faad2913 WatchSource:0}: Error finding container 132c6840955d50a0cfa4916f87c0d3a8faedeb20395873ab3d62e9f2faad2913: Status 404 returned error can't find the container with id 132c6840955d50a0cfa4916f87c0d3a8faedeb20395873ab3d62e9f2faad2913
Apr 24 22:36:01.776902 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:01.776886    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 22:36:02.184566 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:02.184527    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-86cc847c5c-88p77" event={"ID":"2acbce06-e4b4-43e9-ae93-74c97e292076","Type":"ContainerStarted","Data":"132c6840955d50a0cfa4916f87c0d3a8faedeb20395873ab3d62e9f2faad2913"}
Apr 24 22:36:05.195291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:05.195252    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-86cc847c5c-88p77" event={"ID":"2acbce06-e4b4-43e9-ae93-74c97e292076","Type":"ContainerStarted","Data":"17eecf88a33184de2f217f810366a321509a8e4f315b870b08f0cb1aeb5715fe"}
Apr 24 22:36:05.195775 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:05.195358    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:36:05.215621 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:05.215579    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-86cc847c5c-88p77" podStartSLOduration=1.665293428 podStartE2EDuration="4.215565313s" podCreationTimestamp="2026-04-24 22:36:01 +0000 UTC" firstStartedPulling="2026-04-24 22:36:01.77700004 +0000 UTC m=+365.203397548" lastFinishedPulling="2026-04-24 22:36:04.327271912 +0000 UTC m=+367.753669433" observedRunningTime="2026-04-24 22:36:05.213647364 +0000 UTC m=+368.640044895" watchObservedRunningTime="2026-04-24 22:36:05.215565313 +0000 UTC m=+368.641962844"
Apr 24 22:36:11.205856 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:36:11.205826    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-86cc847c5c-88p77"
Apr 24 22:37:26.928987 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:26.928912    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-kw6zd"]
Apr 24 22:37:26.932043 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:26.932025    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kw6zd"
Apr 24 22:37:26.941293 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:26.941271    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kw6zd"]
Apr 24 22:37:27.095210 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:27.095180    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9chw\" (UniqueName: \"kubernetes.io/projected/6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163-kube-api-access-q9chw\") pod \"s3-init-kw6zd\" (UID: \"6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163\") " pod="kserve/s3-init-kw6zd"
Apr 24 22:37:27.196379 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:27.196286    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q9chw\" (UniqueName: \"kubernetes.io/projected/6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163-kube-api-access-q9chw\") pod \"s3-init-kw6zd\" (UID: \"6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163\") " pod="kserve/s3-init-kw6zd"
Apr 24 22:37:27.205621 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:27.205591    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9chw\" (UniqueName: \"kubernetes.io/projected/6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163-kube-api-access-q9chw\") pod \"s3-init-kw6zd\" (UID: \"6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163\") " pod="kserve/s3-init-kw6zd"
Apr 24 22:37:27.251408 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:27.251385    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kw6zd"
Apr 24 22:37:27.370792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:27.370758    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-kw6zd"]
Apr 24 22:37:27.375416 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:37:27.375391    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c1bfbc3_0c19_4a0e_bb63_cfb27dcea163.slice/crio-ae741e716cfcf6f9f035e5f546278ef2593eb109e0d93232cdeb4ec71d0dfbb7 WatchSource:0}: Error finding container ae741e716cfcf6f9f035e5f546278ef2593eb109e0d93232cdeb4ec71d0dfbb7: Status 404 returned error can't find the container with id ae741e716cfcf6f9f035e5f546278ef2593eb109e0d93232cdeb4ec71d0dfbb7
Apr 24 22:37:27.422284 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:27.422257    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kw6zd" event={"ID":"6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163","Type":"ContainerStarted","Data":"ae741e716cfcf6f9f035e5f546278ef2593eb109e0d93232cdeb4ec71d0dfbb7"}
Apr 24 22:37:32.439156 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:32.439120    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kw6zd" event={"ID":"6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163","Type":"ContainerStarted","Data":"e35364ab73ed989aaeea7a939457add2f333e20d2f3905100b4fbaab6a3ea98c"}
Apr 24 22:37:32.458046 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:32.458005    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-kw6zd" podStartSLOduration=2.063596921 podStartE2EDuration="6.457991701s" podCreationTimestamp="2026-04-24 22:37:26 +0000 UTC" firstStartedPulling="2026-04-24 22:37:27.377499601 +0000 UTC m=+450.803897109" lastFinishedPulling="2026-04-24 22:37:31.771894378 +0000 UTC m=+455.198291889" observedRunningTime="2026-04-24 22:37:32.456857061 +0000 UTC m=+455.883254591" watchObservedRunningTime="2026-04-24 22:37:32.457991701 +0000 UTC m=+455.884389232"
Apr 24 22:37:35.448659 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:35.448622    2543 generic.go:358] "Generic (PLEG): container finished" podID="6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163" containerID="e35364ab73ed989aaeea7a939457add2f333e20d2f3905100b4fbaab6a3ea98c" exitCode=0
Apr 24 22:37:35.449007 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:35.448693    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kw6zd" event={"ID":"6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163","Type":"ContainerDied","Data":"e35364ab73ed989aaeea7a939457add2f333e20d2f3905100b4fbaab6a3ea98c"}
Apr 24 22:37:36.585574 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:36.585554    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kw6zd"
Apr 24 22:37:36.663567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:36.663546    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9chw\" (UniqueName: \"kubernetes.io/projected/6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163-kube-api-access-q9chw\") pod \"6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163\" (UID: \"6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163\") "
Apr 24 22:37:36.665716 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:36.665681    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163-kube-api-access-q9chw" (OuterVolumeSpecName: "kube-api-access-q9chw") pod "6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163" (UID: "6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163"). InnerVolumeSpecName "kube-api-access-q9chw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:37:36.764169 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:36.764139    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9chw\" (UniqueName: \"kubernetes.io/projected/6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163-kube-api-access-q9chw\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:37:37.455197 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:37.455118    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-kw6zd" event={"ID":"6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163","Type":"ContainerDied","Data":"ae741e716cfcf6f9f035e5f546278ef2593eb109e0d93232cdeb4ec71d0dfbb7"}
Apr 24 22:37:37.455197 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:37.455150    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-kw6zd"
Apr 24 22:37:37.455412 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:37:37.455152    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae741e716cfcf6f9f035e5f546278ef2593eb109e0d93232cdeb4ec71d0dfbb7"
Apr 24 22:38:13.566303 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.566273    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-djlqt"]
Apr 24 22:38:13.566797 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.566588    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163" containerName="s3-init"
Apr 24 22:38:13.566797 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.566599    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163" containerName="s3-init"
Apr 24 22:38:13.566797 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.566654    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163" containerName="s3-init"
Apr 24 22:38:13.569317 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.569295    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-djlqt"
Apr 24 22:38:13.571609 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.571589    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Apr 24 22:38:13.575037 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.575018    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-djlqt"]
Apr 24 22:38:13.635898 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.635874    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dmsp\" (UniqueName: \"kubernetes.io/projected/c210bd34-254e-4332-9936-f7a2a7ba46fe-kube-api-access-6dmsp\") pod \"s3-tls-init-custom-djlqt\" (UID: \"c210bd34-254e-4332-9936-f7a2a7ba46fe\") " pod="kserve/s3-tls-init-custom-djlqt"
Apr 24 22:38:13.737144 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.737115    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6dmsp\" (UniqueName: \"kubernetes.io/projected/c210bd34-254e-4332-9936-f7a2a7ba46fe-kube-api-access-6dmsp\") pod \"s3-tls-init-custom-djlqt\" (UID: \"c210bd34-254e-4332-9936-f7a2a7ba46fe\") " pod="kserve/s3-tls-init-custom-djlqt"
Apr 24 22:38:13.745439 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.745412    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dmsp\" (UniqueName: \"kubernetes.io/projected/c210bd34-254e-4332-9936-f7a2a7ba46fe-kube-api-access-6dmsp\") pod \"s3-tls-init-custom-djlqt\" (UID: \"c210bd34-254e-4332-9936-f7a2a7ba46fe\") " pod="kserve/s3-tls-init-custom-djlqt"
Apr 24 22:38:13.890623 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:13.890524    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-djlqt"
Apr 24 22:38:14.008968 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:14.008934    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-djlqt"]
Apr 24 22:38:14.012010 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:38:14.011979    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc210bd34_254e_4332_9936_f7a2a7ba46fe.slice/crio-150606ae8c355f43983458ddfdb4017a695a92de58a505a7e835a3811f47a00f WatchSource:0}: Error finding container 150606ae8c355f43983458ddfdb4017a695a92de58a505a7e835a3811f47a00f: Status 404 returned error can't find the container with id 150606ae8c355f43983458ddfdb4017a695a92de58a505a7e835a3811f47a00f
Apr 24 22:38:14.549161 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:14.549125    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-djlqt" event={"ID":"c210bd34-254e-4332-9936-f7a2a7ba46fe","Type":"ContainerStarted","Data":"16193ed225dfe9e1b423b6003f57d65b335d08793693c5011da31be25a9fcc4d"}
Apr 24 22:38:14.549161 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:14.549161    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-djlqt" event={"ID":"c210bd34-254e-4332-9936-f7a2a7ba46fe","Type":"ContainerStarted","Data":"150606ae8c355f43983458ddfdb4017a695a92de58a505a7e835a3811f47a00f"}
Apr 24 22:38:14.564818 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:14.564767    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-djlqt" podStartSLOduration=1.5647534730000001 podStartE2EDuration="1.564753473s" podCreationTimestamp="2026-04-24 22:38:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:38:14.562885084 +0000 UTC m=+497.989282628" watchObservedRunningTime="2026-04-24 22:38:14.564753473 +0000 UTC m=+497.991151003"
Apr 24 22:38:18.566592 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:18.566559    2543 generic.go:358] "Generic (PLEG): container finished" podID="c210bd34-254e-4332-9936-f7a2a7ba46fe" containerID="16193ed225dfe9e1b423b6003f57d65b335d08793693c5011da31be25a9fcc4d" exitCode=0
Apr 24 22:38:18.567049 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:18.566621    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-djlqt" event={"ID":"c210bd34-254e-4332-9936-f7a2a7ba46fe","Type":"ContainerDied","Data":"16193ed225dfe9e1b423b6003f57d65b335d08793693c5011da31be25a9fcc4d"}
Apr 24 22:38:19.682138 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:19.682109    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-djlqt"
Apr 24 22:38:19.786023 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:19.785993    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dmsp\" (UniqueName: \"kubernetes.io/projected/c210bd34-254e-4332-9936-f7a2a7ba46fe-kube-api-access-6dmsp\") pod \"c210bd34-254e-4332-9936-f7a2a7ba46fe\" (UID: \"c210bd34-254e-4332-9936-f7a2a7ba46fe\") "
Apr 24 22:38:19.788142 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:19.788114    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c210bd34-254e-4332-9936-f7a2a7ba46fe-kube-api-access-6dmsp" (OuterVolumeSpecName: "kube-api-access-6dmsp") pod "c210bd34-254e-4332-9936-f7a2a7ba46fe" (UID: "c210bd34-254e-4332-9936-f7a2a7ba46fe"). InnerVolumeSpecName "kube-api-access-6dmsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:38:19.886513 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:19.886446    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dmsp\" (UniqueName: \"kubernetes.io/projected/c210bd34-254e-4332-9936-f7a2a7ba46fe-kube-api-access-6dmsp\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:38:20.573103 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:20.573067    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-djlqt" event={"ID":"c210bd34-254e-4332-9936-f7a2a7ba46fe","Type":"ContainerDied","Data":"150606ae8c355f43983458ddfdb4017a695a92de58a505a7e835a3811f47a00f"}
Apr 24 22:38:20.573103 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:20.573105    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="150606ae8c355f43983458ddfdb4017a695a92de58a505a7e835a3811f47a00f"
Apr 24 22:38:20.573304 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:20.573079    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-djlqt"
Apr 24 22:38:22.793402 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:22.793368    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-k7cd4"]
Apr 24 22:38:22.793806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:22.793662    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c210bd34-254e-4332-9936-f7a2a7ba46fe" containerName="s3-tls-init-custom"
Apr 24 22:38:22.793806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:22.793672    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c210bd34-254e-4332-9936-f7a2a7ba46fe" containerName="s3-tls-init-custom"
Apr 24 22:38:22.793806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:22.793723    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="c210bd34-254e-4332-9936-f7a2a7ba46fe" containerName="s3-tls-init-custom"
Apr 24 22:38:22.796725 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:22.796709    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-k7cd4"
Apr 24 22:38:22.799000 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:22.798979    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Apr 24 22:38:22.801313 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:22.801292    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-k7cd4"]
Apr 24 22:38:22.909439 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:22.909406    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssl7w\" (UniqueName: \"kubernetes.io/projected/f05004c7-2f8d-4ff1-a18e-eb07a140155a-kube-api-access-ssl7w\") pod \"s3-tls-init-serving-k7cd4\" (UID: \"f05004c7-2f8d-4ff1-a18e-eb07a140155a\") " pod="kserve/s3-tls-init-serving-k7cd4"
Apr 24 22:38:23.009888 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:23.009861    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ssl7w\" (UniqueName: \"kubernetes.io/projected/f05004c7-2f8d-4ff1-a18e-eb07a140155a-kube-api-access-ssl7w\") pod \"s3-tls-init-serving-k7cd4\" (UID: \"f05004c7-2f8d-4ff1-a18e-eb07a140155a\") " pod="kserve/s3-tls-init-serving-k7cd4"
Apr 24 22:38:23.019772 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:23.019747    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssl7w\" (UniqueName: \"kubernetes.io/projected/f05004c7-2f8d-4ff1-a18e-eb07a140155a-kube-api-access-ssl7w\") pod \"s3-tls-init-serving-k7cd4\" (UID: \"f05004c7-2f8d-4ff1-a18e-eb07a140155a\") " pod="kserve/s3-tls-init-serving-k7cd4"
Apr 24 22:38:23.113533 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:23.113470    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-k7cd4"
Apr 24 22:38:23.239467 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:23.239432    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-k7cd4"]
Apr 24 22:38:23.241980 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:38:23.241953    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf05004c7_2f8d_4ff1_a18e_eb07a140155a.slice/crio-edf6d4138214d7db42dfd585e4fff8570680b2d1a672028b1fc12c2e3af9b433 WatchSource:0}: Error finding container edf6d4138214d7db42dfd585e4fff8570680b2d1a672028b1fc12c2e3af9b433: Status 404 returned error can't find the container with id edf6d4138214d7db42dfd585e4fff8570680b2d1a672028b1fc12c2e3af9b433
Apr 24 22:38:23.583497 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:23.583449    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-k7cd4" event={"ID":"f05004c7-2f8d-4ff1-a18e-eb07a140155a","Type":"ContainerStarted","Data":"5c307ea5c23c04c9fbe325bd5eff60da02b61c5b00286836f8b1dc4e2c4474c8"}
Apr 24 22:38:23.583497 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:23.583494    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-k7cd4" event={"ID":"f05004c7-2f8d-4ff1-a18e-eb07a140155a","Type":"ContainerStarted","Data":"edf6d4138214d7db42dfd585e4fff8570680b2d1a672028b1fc12c2e3af9b433"}
Apr 24 22:38:23.600197 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:23.600155    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-k7cd4" podStartSLOduration=1.600143246 podStartE2EDuration="1.600143246s" podCreationTimestamp="2026-04-24 22:38:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:38:23.59850165 +0000 UTC m=+507.024899181" watchObservedRunningTime="2026-04-24 22:38:23.600143246 +0000 UTC m=+507.026540777"
Apr 24 22:38:27.594900 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:27.594876    2543 generic.go:358] "Generic (PLEG): container finished" podID="f05004c7-2f8d-4ff1-a18e-eb07a140155a" containerID="5c307ea5c23c04c9fbe325bd5eff60da02b61c5b00286836f8b1dc4e2c4474c8" exitCode=0
Apr 24 22:38:27.595178 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:27.594932    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-k7cd4" event={"ID":"f05004c7-2f8d-4ff1-a18e-eb07a140155a","Type":"ContainerDied","Data":"5c307ea5c23c04c9fbe325bd5eff60da02b61c5b00286836f8b1dc4e2c4474c8"}
Apr 24 22:38:28.717424 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:28.717402    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-k7cd4"
Apr 24 22:38:28.853948 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:28.853877    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssl7w\" (UniqueName: \"kubernetes.io/projected/f05004c7-2f8d-4ff1-a18e-eb07a140155a-kube-api-access-ssl7w\") pod \"f05004c7-2f8d-4ff1-a18e-eb07a140155a\" (UID: \"f05004c7-2f8d-4ff1-a18e-eb07a140155a\") "
Apr 24 22:38:28.855886 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:28.855862    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05004c7-2f8d-4ff1-a18e-eb07a140155a-kube-api-access-ssl7w" (OuterVolumeSpecName: "kube-api-access-ssl7w") pod "f05004c7-2f8d-4ff1-a18e-eb07a140155a" (UID: "f05004c7-2f8d-4ff1-a18e-eb07a140155a"). InnerVolumeSpecName "kube-api-access-ssl7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:38:28.954520 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:28.954494    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssl7w\" (UniqueName: \"kubernetes.io/projected/f05004c7-2f8d-4ff1-a18e-eb07a140155a-kube-api-access-ssl7w\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:38:29.601100 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:29.601064    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-k7cd4" event={"ID":"f05004c7-2f8d-4ff1-a18e-eb07a140155a","Type":"ContainerDied","Data":"edf6d4138214d7db42dfd585e4fff8570680b2d1a672028b1fc12c2e3af9b433"}
Apr 24 22:38:29.601100 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:29.601100    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-k7cd4"
Apr 24 22:38:29.601291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:29.601101    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf6d4138214d7db42dfd585e4fff8570680b2d1a672028b1fc12c2e3af9b433"
Apr 24 22:38:38.551999 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.551917    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"]
Apr 24 22:38:38.552462 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.552361    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f05004c7-2f8d-4ff1-a18e-eb07a140155a" containerName="s3-tls-init-serving"
Apr 24 22:38:38.552462 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.552380    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f05004c7-2f8d-4ff1-a18e-eb07a140155a" containerName="s3-tls-init-serving"
Apr 24 22:38:38.552462 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.552451    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="f05004c7-2f8d-4ff1-a18e-eb07a140155a" containerName="s3-tls-init-serving"
Apr 24 22:38:38.555778 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.555755    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.558267 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.558241    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Apr 24 22:38:38.558495 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.558478    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 24 22:38:38.559448 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.559418    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Apr 24 22:38:38.559524 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.559452    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 24 22:38:38.559584 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.559552    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-hkqwb\""
Apr 24 22:38:38.566956 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.566929    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"]
Apr 24 22:38:38.622013 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.621982    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61a98e46-df8c-4fd2-9f35-6b6262327e5a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.622117 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.622020    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8q8q\" (UniqueName: \"kubernetes.io/projected/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kube-api-access-k8q8q\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.622117 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.622059    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61a98e46-df8c-4fd2-9f35-6b6262327e5a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.622117 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.622097    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.722577 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.722546    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61a98e46-df8c-4fd2-9f35-6b6262327e5a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.722687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.722592    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k8q8q\" (UniqueName: \"kubernetes.io/projected/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kube-api-access-k8q8q\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.722687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.722628    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61a98e46-df8c-4fd2-9f35-6b6262327e5a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.722687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.722682    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.723023 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.723002    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.723200 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.723182    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61a98e46-df8c-4fd2-9f35-6b6262327e5a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.725104 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.725088    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61a98e46-df8c-4fd2-9f35-6b6262327e5a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.730934 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.730913    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8q8q\" (UniqueName: \"kubernetes.io/projected/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kube-api-access-k8q8q\") pod \"isvc-sklearn-batcher-predictor-587b88589b-bk975\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.866725 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.866668    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:38:38.986001 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:38.985937    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"]
Apr 24 22:38:38.988709 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:38:38.988681    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a98e46_df8c_4fd2_9f35_6b6262327e5a.slice/crio-ca577a64989ca7e689f2070cb6164873f37b831750590ecd31adc56d17249060 WatchSource:0}: Error finding container ca577a64989ca7e689f2070cb6164873f37b831750590ecd31adc56d17249060: Status 404 returned error can't find the container with id ca577a64989ca7e689f2070cb6164873f37b831750590ecd31adc56d17249060
Apr 24 22:38:39.629283 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:39.629230    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerStarted","Data":"ca577a64989ca7e689f2070cb6164873f37b831750590ecd31adc56d17249060"}
Apr 24 22:38:42.641033 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:42.640944    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerStarted","Data":"dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42"}
Apr 24 22:38:46.653474 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:46.653441    2543 generic.go:358] "Generic (PLEG): container finished" podID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerID="dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42" exitCode=0
Apr 24 22:38:46.653845 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:38:46.653517    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerDied","Data":"dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42"}
Apr 24 22:39:00.706727 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:00.706690    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerStarted","Data":"33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452"}
Apr 24 22:39:02.716914 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:02.716875    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerStarted","Data":"73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d"}
Apr 24 22:39:04.724484 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:04.724444    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerStarted","Data":"350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb"}
Apr 24 22:39:04.724830 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:04.724734    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:39:04.724830 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:04.724769    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:39:04.724830 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:04.724782    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:39:04.726264 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:04.726232    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:39:04.726920 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:04.726899    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:39:04.745377 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:04.745315    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podStartSLOduration=1.15985641 podStartE2EDuration="26.745303889s" podCreationTimestamp="2026-04-24 22:38:38 +0000 UTC" firstStartedPulling="2026-04-24 22:38:38.99060365 +0000 UTC m=+522.417001158" lastFinishedPulling="2026-04-24 22:39:04.576051128 +0000 UTC m=+548.002448637" observedRunningTime="2026-04-24 22:39:04.743616601 +0000 UTC m=+548.170014131" watchObservedRunningTime="2026-04-24 22:39:04.745303889 +0000 UTC m=+548.171701421"
Apr 24 22:39:05.727132 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:05.727089    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:39:05.727614 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:05.727582    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:39:10.731590 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:10.731558    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:39:10.732247 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:10.732210    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:39:10.732401 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:10.732328    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:39:20.732170 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:20.732125    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:39:20.732632 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:20.732609    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:39:30.732674 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:30.732634    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:39:30.733117 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:30.733009    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:39:40.733004 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:40.732963    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:39:40.733469 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:40.733447    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:39:50.732080 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:50.732037    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:39:50.732576 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:50.732509    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:39:56.999250 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:56.999220    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:39:57.000952 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:39:57.000931    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:40:00.732518 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:00.732485    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:40:00.733018 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:00.732997    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:40:13.543910 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.543833    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"]
Apr 24 22:40:13.544424 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.544247    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" containerID="cri-o://33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452" gracePeriod=30
Apr 24 22:40:13.544424 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.544408    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy" containerID="cri-o://73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d" gracePeriod=30
Apr 24 22:40:13.544562 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.544488    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" containerID="cri-o://350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb" gracePeriod=30
Apr 24 22:40:13.645832 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.645801    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"]
Apr 24 22:40:13.649530 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.649506    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.651827 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.651809    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Apr 24 22:40:13.651926 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.651828    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Apr 24 22:40:13.658537 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.658518    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"]
Apr 24 22:40:13.750407 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.750381    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795d7a1d-84da-48b7-a711-467bca095ba5-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.750524 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.750413    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795d7a1d-84da-48b7-a711-467bca095ba5-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.750524 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.750449    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrs8\" (UniqueName: \"kubernetes.io/projected/795d7a1d-84da-48b7-a711-467bca095ba5-kube-api-access-vmrs8\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.750524 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.750484    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795d7a1d-84da-48b7-a711-467bca095ba5-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.851784 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.851728    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795d7a1d-84da-48b7-a711-467bca095ba5-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.851784 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.851757    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795d7a1d-84da-48b7-a711-467bca095ba5-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.851958 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.851787    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrs8\" (UniqueName: \"kubernetes.io/projected/795d7a1d-84da-48b7-a711-467bca095ba5-kube-api-access-vmrs8\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.851958 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.851819    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795d7a1d-84da-48b7-a711-467bca095ba5-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.852218 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.852202    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795d7a1d-84da-48b7-a711-467bca095ba5-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.852506 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.852489    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795d7a1d-84da-48b7-a711-467bca095ba5-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.854147 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.854131    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795d7a1d-84da-48b7-a711-467bca095ba5-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.859190 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.859165    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrs8\" (UniqueName: \"kubernetes.io/projected/795d7a1d-84da-48b7-a711-467bca095ba5-kube-api-access-vmrs8\") pod \"isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:13.920007 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.919982    2543 generic.go:358] "Generic (PLEG): container finished" podID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerID="73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d" exitCode=2
Apr 24 22:40:13.920111 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.920061    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerDied","Data":"73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d"}
Apr 24 22:40:13.961149 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:13.961125    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:14.086037 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:14.085921    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"]
Apr 24 22:40:14.088731 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:40:14.088698    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795d7a1d_84da_48b7_a711_467bca095ba5.slice/crio-ab2dd0b91efd17b879e36f5d4e33e912772dc28edcb033b9d23d566d122b3b92 WatchSource:0}: Error finding container ab2dd0b91efd17b879e36f5d4e33e912772dc28edcb033b9d23d566d122b3b92: Status 404 returned error can't find the container with id ab2dd0b91efd17b879e36f5d4e33e912772dc28edcb033b9d23d566d122b3b92
Apr 24 22:40:14.925318 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:14.925283    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerStarted","Data":"fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be"}
Apr 24 22:40:14.925744 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:14.925327    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerStarted","Data":"ab2dd0b91efd17b879e36f5d4e33e912772dc28edcb033b9d23d566d122b3b92"}
Apr 24 22:40:15.727752 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:15.727692    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Apr 24 22:40:17.935770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:17.935743    2543 generic.go:358] "Generic (PLEG): container finished" podID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerID="33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452" exitCode=0
Apr 24 22:40:17.936114 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:17.935808    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerDied","Data":"33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452"}
Apr 24 22:40:17.937130 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:17.937108    2543 generic.go:358] "Generic (PLEG): container finished" podID="795d7a1d-84da-48b7-a711-467bca095ba5" containerID="fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be" exitCode=0
Apr 24 22:40:17.937224 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:17.937139    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerDied","Data":"fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be"}
Apr 24 22:40:18.942153 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:18.942118    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerStarted","Data":"d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b"}
Apr 24 22:40:18.942153 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:18.942160    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerStarted","Data":"591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d"}
Apr 24 22:40:18.942774 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:18.942174    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerStarted","Data":"844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a"}
Apr 24 22:40:18.942774 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:18.942436    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:18.973293 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:18.973254    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podStartSLOduration=5.973242737 podStartE2EDuration="5.973242737s" podCreationTimestamp="2026-04-24 22:40:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:40:18.971557652 +0000 UTC m=+622.397955181" watchObservedRunningTime="2026-04-24 22:40:18.973242737 +0000 UTC m=+622.399640264"
Apr 24 22:40:19.944899 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:19.944865    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:19.945245 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:19.944907    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:19.946274 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:19.946246    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:40:19.946995 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:19.946966    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:40:20.727940 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:20.727902    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Apr 24 22:40:20.732876 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:20.732849    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:40:20.733509 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:20.733489    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:40:20.948117 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:20.948084    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:40:20.948481 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:20.948416    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:40:25.727251 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:25.727204    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Apr 24 22:40:25.727639 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:25.727355    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:40:25.953367 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:25.953325    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:40:25.953912 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:25.953883    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:40:25.954110 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:25.954089    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:40:30.728082 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:30.728037    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Apr 24 22:40:30.732867 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:30.732842    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:40:30.733404 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:30.733383    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:40:35.728323 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:35.728276    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Apr 24 22:40:35.954772 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:35.954727    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:40:35.955183 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:35.955158    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:40:40.728078 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:40.728040    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Apr 24 22:40:40.732403 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:40.732379    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Apr 24 22:40:40.732500 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:40.732488    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:40:40.733981 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:40.733956    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:40:40.734059 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:40.734052    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:40:43.679549 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.679073    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:40:43.785144 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.785115    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61a98e46-df8c-4fd2-9f35-6b6262327e5a-proxy-tls\") pod \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") "
Apr 24 22:40:43.785291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.785149    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kserve-provision-location\") pod \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") "
Apr 24 22:40:43.785384 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.785364    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61a98e46-df8c-4fd2-9f35-6b6262327e5a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") "
Apr 24 22:40:43.785446 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.785432    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8q8q\" (UniqueName: \"kubernetes.io/projected/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kube-api-access-k8q8q\") pod \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\" (UID: \"61a98e46-df8c-4fd2-9f35-6b6262327e5a\") "
Apr 24 22:40:43.785484 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.785453    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61a98e46-df8c-4fd2-9f35-6b6262327e5a" (UID: "61a98e46-df8c-4fd2-9f35-6b6262327e5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:40:43.785631 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.785613    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:40:43.785739 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.785694    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a98e46-df8c-4fd2-9f35-6b6262327e5a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "61a98e46-df8c-4fd2-9f35-6b6262327e5a" (UID: "61a98e46-df8c-4fd2-9f35-6b6262327e5a"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:40:43.787390 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.787362    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a98e46-df8c-4fd2-9f35-6b6262327e5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61a98e46-df8c-4fd2-9f35-6b6262327e5a" (UID: "61a98e46-df8c-4fd2-9f35-6b6262327e5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:40:43.787533 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.787514    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kube-api-access-k8q8q" (OuterVolumeSpecName: "kube-api-access-k8q8q") pod "61a98e46-df8c-4fd2-9f35-6b6262327e5a" (UID: "61a98e46-df8c-4fd2-9f35-6b6262327e5a"). InnerVolumeSpecName "kube-api-access-k8q8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:40:43.886221 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.886166    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8q8q\" (UniqueName: \"kubernetes.io/projected/61a98e46-df8c-4fd2-9f35-6b6262327e5a-kube-api-access-k8q8q\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:40:43.886221 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.886188    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61a98e46-df8c-4fd2-9f35-6b6262327e5a-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:40:43.886221 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:43.886200    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61a98e46-df8c-4fd2-9f35-6b6262327e5a-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:40:44.019620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.019582    2543 generic.go:358] "Generic (PLEG): container finished" podID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerID="350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb" exitCode=0
Apr 24 22:40:44.019773 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.019660    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerDied","Data":"350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb"}
Apr 24 22:40:44.019773 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.019669    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"
Apr 24 22:40:44.019773 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.019687    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975" event={"ID":"61a98e46-df8c-4fd2-9f35-6b6262327e5a","Type":"ContainerDied","Data":"ca577a64989ca7e689f2070cb6164873f37b831750590ecd31adc56d17249060"}
Apr 24 22:40:44.019773 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.019703    2543 scope.go:117] "RemoveContainer" containerID="350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb"
Apr 24 22:40:44.028193 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.028174    2543 scope.go:117] "RemoveContainer" containerID="73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d"
Apr 24 22:40:44.035357 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.035322    2543 scope.go:117] "RemoveContainer" containerID="33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452"
Apr 24 22:40:44.041500 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.041478    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"]
Apr 24 22:40:44.042190 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.042171    2543 scope.go:117] "RemoveContainer" containerID="dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42"
Apr 24 22:40:44.044619 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.044600    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-587b88589b-bk975"]
Apr 24 22:40:44.049024 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.049006    2543 scope.go:117] "RemoveContainer" containerID="350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb"
Apr 24 22:40:44.049256 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:40:44.049238    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb\": container with ID starting with 350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb not found: ID does not exist" containerID="350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb"
Apr 24 22:40:44.049322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.049268    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb"} err="failed to get container status \"350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb\": rpc error: code = NotFound desc = could not find container \"350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb\": container with ID starting with 350b6a2a5d3db469fbfed5a9f21a2898cde3f13b72888a2a7ef6d371dfdd2aeb not found: ID does not exist"
Apr 24 22:40:44.049322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.049293    2543 scope.go:117] "RemoveContainer" containerID="73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d"
Apr 24 22:40:44.049588 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:40:44.049570    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d\": container with ID starting with 73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d not found: ID does not exist" containerID="73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d"
Apr 24 22:40:44.049640 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.049593    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d"} err="failed to get container status \"73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d\": rpc error: code = NotFound desc = could not find container \"73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d\": container with ID starting with 73d448f103680cce4deb95f3796191b26fcab3c3a54e35b97b92582fe465692d not found: ID does not exist"
Apr 24 22:40:44.049640 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.049609    2543 scope.go:117] "RemoveContainer" containerID="33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452"
Apr 24 22:40:44.049837 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:40:44.049819    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452\": container with ID starting with 33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452 not found: ID does not exist" containerID="33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452"
Apr 24 22:40:44.049901 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.049848    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452"} err="failed to get container status \"33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452\": rpc error: code = NotFound desc = could not find container \"33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452\": container with ID starting with 33839eb182962efface43ac5726ada01e940878b0a7aef89995b1b15c92a9452 not found: ID does not exist"
Apr 24 22:40:44.049901 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.049870    2543 scope.go:117] "RemoveContainer" containerID="dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42"
Apr 24 22:40:44.050101 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:40:44.050086    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42\": container with ID starting with dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42 not found: ID does not exist" containerID="dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42"
Apr 24 22:40:44.050138 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:44.050105    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42"} err="failed to get container status \"dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42\": rpc error: code = NotFound desc = could not find container \"dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42\": container with ID starting with dd769c2550de0574f6543a15823a2cddeb0ec81de82f14ddf735dbc85bc2db42 not found: ID does not exist"
Apr 24 22:40:45.054632 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:45.054598    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" path="/var/lib/kubelet/pods/61a98e46-df8c-4fd2-9f35-6b6262327e5a/volumes"
Apr 24 22:40:45.954755 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:45.954712    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:40:45.955157 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:45.955131    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:40:55.953991 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:55.953951    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:40:55.954483 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:40:55.954422    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:41:05.953821 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:05.953770    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:41:05.954282 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:05.954198    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:41:15.954441 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:15.954394    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:41:15.954853 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:15.954831    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:41:25.954415 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:25.954388    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:41:25.954839 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:25.954450    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:41:38.711425 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.711319    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"]
Apr 24 22:41:38.711828 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.711695    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" containerID="cri-o://844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a" gracePeriod=30
Apr 24 22:41:38.711828 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.711763    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" containerID="cri-o://d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b" gracePeriod=30
Apr 24 22:41:38.711981 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.711813    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy" containerID="cri-o://591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d" gracePeriod=30
Apr 24 22:41:38.772974 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.772946    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"]
Apr 24 22:41:38.773251 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773238    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy"
Apr 24 22:41:38.773295 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773253    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy"
Apr 24 22:41:38.773295 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773268    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent"
Apr 24 22:41:38.773295 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773273    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent"
Apr 24 22:41:38.773295 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773287    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="storage-initializer"
Apr 24 22:41:38.773295 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773293    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="storage-initializer"
Apr 24 22:41:38.773466 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773313    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container"
Apr 24 22:41:38.773466 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773320    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container"
Apr 24 22:41:38.773466 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773379    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="agent"
Apr 24 22:41:38.773466 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773393    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kserve-container"
Apr 24 22:41:38.773466 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.773401    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="61a98e46-df8c-4fd2-9f35-6b6262327e5a" containerName="kube-rbac-proxy"
Apr 24 22:41:38.776692 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.776676    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:38.778995 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.778980    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Apr 24 22:41:38.779079 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.779014    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Apr 24 22:41:38.787016 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.786997    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"]
Apr 24 22:41:38.863294 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.863269    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9zt\" (UniqueName: \"kubernetes.io/projected/d81e7f17-a9f6-4c1a-bbef-23886621e642-kube-api-access-qn9zt\") pod \"message-dumper-predictor-c7d86bcbd-p9nkd\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:38.863433 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.863327    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81e7f17-a9f6-4c1a-bbef-23886621e642-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-p9nkd\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:38.863433 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.863376    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81e7f17-a9f6-4c1a-bbef-23886621e642-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-p9nkd\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:38.964457 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.964394    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9zt\" (UniqueName: \"kubernetes.io/projected/d81e7f17-a9f6-4c1a-bbef-23886621e642-kube-api-access-qn9zt\") pod \"message-dumper-predictor-c7d86bcbd-p9nkd\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:38.964457 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.964440    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81e7f17-a9f6-4c1a-bbef-23886621e642-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-p9nkd\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:38.964623 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.964468    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81e7f17-a9f6-4c1a-bbef-23886621e642-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-p9nkd\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:38.965145 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.965121    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81e7f17-a9f6-4c1a-bbef-23886621e642-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-p9nkd\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:38.966943 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.966923    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81e7f17-a9f6-4c1a-bbef-23886621e642-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-p9nkd\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:38.973158 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:38.973134    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9zt\" (UniqueName: \"kubernetes.io/projected/d81e7f17-a9f6-4c1a-bbef-23886621e642-kube-api-access-qn9zt\") pod \"message-dumper-predictor-c7d86bcbd-p9nkd\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:39.086806 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:39.086783    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:39.173434 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:39.173404    2543 generic.go:358] "Generic (PLEG): container finished" podID="795d7a1d-84da-48b7-a711-467bca095ba5" containerID="591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d" exitCode=2
Apr 24 22:41:39.173568 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:39.173478    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerDied","Data":"591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d"}
Apr 24 22:41:39.207405 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:39.207381    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"]
Apr 24 22:41:39.210570 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:41:39.210544    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81e7f17_a9f6_4c1a_bbef_23886621e642.slice/crio-3a42351585c7f3420ae410936161099281230aef270313f157c9d8d79a242816 WatchSource:0}: Error finding container 3a42351585c7f3420ae410936161099281230aef270313f157c9d8d79a242816: Status 404 returned error can't find the container with id 3a42351585c7f3420ae410936161099281230aef270313f157c9d8d79a242816
Apr 24 22:41:39.212176 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:39.212158    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 22:41:40.177624 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:40.177602    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd" event={"ID":"d81e7f17-a9f6-4c1a-bbef-23886621e642","Type":"ContainerStarted","Data":"3a42351585c7f3420ae410936161099281230aef270313f157c9d8d79a242816"}
Apr 24 22:41:40.948267 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:40.948227    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Apr 24 22:41:41.181837 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:41.181805    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd" event={"ID":"d81e7f17-a9f6-4c1a-bbef-23886621e642","Type":"ContainerStarted","Data":"888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13"}
Apr 24 22:41:41.182224 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:41.181845    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd" event={"ID":"d81e7f17-a9f6-4c1a-bbef-23886621e642","Type":"ContainerStarted","Data":"b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca"}
Apr 24 22:41:41.182224 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:41.182021    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:41.182224 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:41.182152    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:41.183687 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:41.183669    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:41.200547 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:41.200473    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd" podStartSLOduration=2.242209772 podStartE2EDuration="3.200460064s" podCreationTimestamp="2026-04-24 22:41:38 +0000 UTC" firstStartedPulling="2026-04-24 22:41:39.212308636 +0000 UTC m=+702.638706148" lastFinishedPulling="2026-04-24 22:41:40.170558928 +0000 UTC m=+703.596956440" observedRunningTime="2026-04-24 22:41:41.199583763 +0000 UTC m=+704.625981293" watchObservedRunningTime="2026-04-24 22:41:41.200460064 +0000 UTC m=+704.626857594"
Apr 24 22:41:43.188960 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:43.188932    2543 generic.go:358] "Generic (PLEG): container finished" podID="795d7a1d-84da-48b7-a711-467bca095ba5" containerID="844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a" exitCode=0
Apr 24 22:41:43.189403 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:43.189005    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerDied","Data":"844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a"}
Apr 24 22:41:45.948210 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:45.948168    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Apr 24 22:41:45.954527 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:45.954503    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:41:45.954821 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:45.954798    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:41:48.193810 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.193623    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:41:48.816738 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.816712    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"]
Apr 24 22:41:48.820266 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.820248    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.822500 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.822484    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Apr 24 22:41:48.822500 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.822493    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Apr 24 22:41:48.829961 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.829937    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"]
Apr 24 22:41:48.834588 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.834566    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-proxy-tls\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.834668 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.834597    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlh9\" (UniqueName: \"kubernetes.io/projected/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kube-api-access-hhlh9\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.834709 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.834665    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.834709 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.834687    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kserve-provision-location\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.935373 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.935328    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.935507 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.935386    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kserve-provision-location\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.935507 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.935438    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-proxy-tls\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.935507 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.935454    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlh9\" (UniqueName: \"kubernetes.io/projected/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kube-api-access-hhlh9\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.935672 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:41:48.935591    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Apr 24 22:41:48.935672 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:41:48.935659    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-proxy-tls podName:2d2bd2bc-d9d8-4210-82c8-5e215265e1eb nodeName:}" failed. No retries permitted until 2026-04-24 22:41:49.43564223 +0000 UTC m=+712.862039738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-proxy-tls") pod "isvc-logger-predictor-7b48948c-mjqrd" (UID: "2d2bd2bc-d9d8-4210-82c8-5e215265e1eb") : secret "isvc-logger-predictor-serving-cert" not found
Apr 24 22:41:48.935853 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.935834    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kserve-provision-location\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.936050 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.936030    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:48.943848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:48.943829    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlh9\" (UniqueName: \"kubernetes.io/projected/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kube-api-access-hhlh9\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:49.439732 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:49.439700    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-proxy-tls\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:49.442230 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:49.442207    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-proxy-tls\") pod \"isvc-logger-predictor-7b48948c-mjqrd\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:49.730731 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:49.730698    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:49.856431 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:49.856405    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"]
Apr 24 22:41:49.859605 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:41:49.859573    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2bd2bc_d9d8_4210_82c8_5e215265e1eb.slice/crio-7d14a94ab95b52ac4afdbee85eb680ea901d27e265382b21160515fa702682ce WatchSource:0}: Error finding container 7d14a94ab95b52ac4afdbee85eb680ea901d27e265382b21160515fa702682ce: Status 404 returned error can't find the container with id 7d14a94ab95b52ac4afdbee85eb680ea901d27e265382b21160515fa702682ce
Apr 24 22:41:50.210790 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:50.210758    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerStarted","Data":"9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb"}
Apr 24 22:41:50.210790 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:50.210796    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerStarted","Data":"7d14a94ab95b52ac4afdbee85eb680ea901d27e265382b21160515fa702682ce"}
Apr 24 22:41:50.948424 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:50.948386    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Apr 24 22:41:50.948750 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:50.948538    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:41:54.223210 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:54.223178    2543 generic.go:358] "Generic (PLEG): container finished" podID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerID="9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb" exitCode=0
Apr 24 22:41:54.223612 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:54.223247    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerDied","Data":"9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb"}
Apr 24 22:41:55.231621 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.231585    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerStarted","Data":"bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383"}
Apr 24 22:41:55.231979 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.231632    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerStarted","Data":"cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a"}
Apr 24 22:41:55.231979 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.231643    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerStarted","Data":"95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7"}
Apr 24 22:41:55.231979 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.231924    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:55.232124 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.232053    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:55.233273 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.233248    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:41:55.251522 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.251476    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podStartSLOduration=7.251463858 podStartE2EDuration="7.251463858s" podCreationTimestamp="2026-04-24 22:41:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:41:55.250112323 +0000 UTC m=+718.676509853" watchObservedRunningTime="2026-04-24 22:41:55.251463858 +0000 UTC m=+718.677861388"
Apr 24 22:41:55.949026 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.948987    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Apr 24 22:41:55.953811 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.953777    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:41:55.954144 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:55.954121    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:41:56.235279 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:56.235245    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:41:56.235748 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:56.235333    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:41:56.236159 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:56.236136    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:41:57.237549 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:57.237507    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:41:57.237914 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:41:57.237805    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:42:00.948977 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:00.948930    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Apr 24 22:42:02.241491 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:02.241458    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:42:02.242099 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:02.242067    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:42:02.242534 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:02.242511    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:42:05.948214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:05.948167    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Apr 24 22:42:05.954762 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:05.954732    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:5000: connect: connection refused"
Apr 24 22:42:05.954868 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:05.954855    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:42:05.955099 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:05.955071    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:42:05.955205 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:05.955193    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:42:08.855964 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.855943    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:42:08.883930 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.883906    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795d7a1d-84da-48b7-a711-467bca095ba5-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"795d7a1d-84da-48b7-a711-467bca095ba5\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") "
Apr 24 22:42:08.884070 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.883940    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795d7a1d-84da-48b7-a711-467bca095ba5-proxy-tls\") pod \"795d7a1d-84da-48b7-a711-467bca095ba5\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") "
Apr 24 22:42:08.884070 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.883999    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795d7a1d-84da-48b7-a711-467bca095ba5-kserve-provision-location\") pod \"795d7a1d-84da-48b7-a711-467bca095ba5\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") "
Apr 24 22:42:08.884070 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.884049    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmrs8\" (UniqueName: \"kubernetes.io/projected/795d7a1d-84da-48b7-a711-467bca095ba5-kube-api-access-vmrs8\") pod \"795d7a1d-84da-48b7-a711-467bca095ba5\" (UID: \"795d7a1d-84da-48b7-a711-467bca095ba5\") "
Apr 24 22:42:08.884304 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.884265    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795d7a1d-84da-48b7-a711-467bca095ba5-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "795d7a1d-84da-48b7-a711-467bca095ba5" (UID: "795d7a1d-84da-48b7-a711-467bca095ba5"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:42:08.884404 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.884328    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d7a1d-84da-48b7-a711-467bca095ba5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "795d7a1d-84da-48b7-a711-467bca095ba5" (UID: "795d7a1d-84da-48b7-a711-467bca095ba5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:42:08.886177 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.886154    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795d7a1d-84da-48b7-a711-467bca095ba5-kube-api-access-vmrs8" (OuterVolumeSpecName: "kube-api-access-vmrs8") pod "795d7a1d-84da-48b7-a711-467bca095ba5" (UID: "795d7a1d-84da-48b7-a711-467bca095ba5"). InnerVolumeSpecName "kube-api-access-vmrs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:42:08.886282 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.886207    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/795d7a1d-84da-48b7-a711-467bca095ba5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "795d7a1d-84da-48b7-a711-467bca095ba5" (UID: "795d7a1d-84da-48b7-a711-467bca095ba5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:42:08.984611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.984542    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/795d7a1d-84da-48b7-a711-467bca095ba5-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:42:08.984611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.984566    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrs8\" (UniqueName: \"kubernetes.io/projected/795d7a1d-84da-48b7-a711-467bca095ba5-kube-api-access-vmrs8\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:42:08.984611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.984577    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/795d7a1d-84da-48b7-a711-467bca095ba5-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:42:08.984611 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:08.984588    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/795d7a1d-84da-48b7-a711-467bca095ba5-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:42:09.273277 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.273190    2543 generic.go:358] "Generic (PLEG): container finished" podID="795d7a1d-84da-48b7-a711-467bca095ba5" containerID="d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b" exitCode=0
Apr 24 22:42:09.273277 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.273251    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerDied","Data":"d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b"}
Apr 24 22:42:09.273476 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.273286    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69" event={"ID":"795d7a1d-84da-48b7-a711-467bca095ba5","Type":"ContainerDied","Data":"ab2dd0b91efd17b879e36f5d4e33e912772dc28edcb033b9d23d566d122b3b92"}
Apr 24 22:42:09.273476 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.273292    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"
Apr 24 22:42:09.273476 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.273308    2543 scope.go:117] "RemoveContainer" containerID="d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b"
Apr 24 22:42:09.281209 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.281179    2543 scope.go:117] "RemoveContainer" containerID="591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d"
Apr 24 22:42:09.288313 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.288295    2543 scope.go:117] "RemoveContainer" containerID="844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a"
Apr 24 22:42:09.291781 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.291760    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"]
Apr 24 22:42:09.295533 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.295518    2543 scope.go:117] "RemoveContainer" containerID="fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be"
Apr 24 22:42:09.297737 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.297717    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-58c84cb6b9-c7f69"]
Apr 24 22:42:09.302562 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.302544    2543 scope.go:117] "RemoveContainer" containerID="d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b"
Apr 24 22:42:09.302805 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:42:09.302787    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b\": container with ID starting with d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b not found: ID does not exist" containerID="d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b"
Apr 24 22:42:09.302848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.302813    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b"} err="failed to get container status \"d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b\": rpc error: code = NotFound desc = could not find container \"d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b\": container with ID starting with d67800a9f3329c7fb943f3a27c8331ef5beea68e4feee73a511038af30914e4b not found: ID does not exist"
Apr 24 22:42:09.302848 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.302829    2543 scope.go:117] "RemoveContainer" containerID="591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d"
Apr 24 22:42:09.303049 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:42:09.303031    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d\": container with ID starting with 591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d not found: ID does not exist" containerID="591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d"
Apr 24 22:42:09.303088 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.303054    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d"} err="failed to get container status \"591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d\": rpc error: code = NotFound desc = could not find container \"591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d\": container with ID starting with 591804f2166b88c30939d3d9f41aa53c7a65772da06559a21f6d85ab68c3739d not found: ID does not exist"
Apr 24 22:42:09.303088 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.303067    2543 scope.go:117] "RemoveContainer" containerID="844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a"
Apr 24 22:42:09.303271 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:42:09.303253    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a\": container with ID starting with 844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a not found: ID does not exist" containerID="844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a"
Apr 24 22:42:09.303317 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.303281    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a"} err="failed to get container status \"844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a\": rpc error: code = NotFound desc = could not find container \"844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a\": container with ID starting with 844f82446845d96cdfb2f157a5ff32cf2eba1b8cbc26ef3184ddb85b2420b87a not found: ID does not exist"
Apr 24 22:42:09.303317 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.303305    2543 scope.go:117] "RemoveContainer" containerID="fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be"
Apr 24 22:42:09.303539 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:42:09.303524    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be\": container with ID starting with fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be not found: ID does not exist" containerID="fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be"
Apr 24 22:42:09.303580 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:09.303542    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be"} err="failed to get container status \"fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be\": rpc error: code = NotFound desc = could not find container \"fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be\": container with ID starting with fdc0c94fb8e29900238638c693f58b73d1607085ad6ea7c9108deba5154ab5be not found: ID does not exist"
Apr 24 22:42:11.055119 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:11.055085    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" path="/var/lib/kubelet/pods/795d7a1d-84da-48b7-a711-467bca095ba5/volumes"
Apr 24 22:42:12.242547 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:12.242497    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:42:12.242954 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:12.242908    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:42:22.241985 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:22.241936    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:42:22.242399 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:22.242362    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:42:32.242547 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:32.242508    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:42:32.243023 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:32.242992    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:42:42.242666 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:42.242609    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:42:42.243112 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:42.243045    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:42:52.242774 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:52.242731    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:42:52.243291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:42:52.243268    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:43:02.242497 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:02.242469    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:43:02.242926 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:02.242630    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:43:13.839294 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:13.839224    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-c7d86bcbd-p9nkd_d81e7f17-a9f6-4c1a-bbef-23886621e642/kserve-container/0.log"
Apr 24 22:43:14.017752 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.017718    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"]
Apr 24 22:43:14.018224 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.018079    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" containerID="cri-o://95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7" gracePeriod=30
Apr 24 22:43:14.018224 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.018134    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy" containerID="cri-o://cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a" gracePeriod=30
Apr 24 22:43:14.018224 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.018130    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" containerID="cri-o://bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383" gracePeriod=30
Apr 24 22:43:14.078682 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.078656    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"]
Apr 24 22:43:14.079013 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.078999    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="storage-initializer"
Apr 24 22:43:14.079073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079015    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="storage-initializer"
Apr 24 22:43:14.079073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079036    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy"
Apr 24 22:43:14.079073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079046    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy"
Apr 24 22:43:14.079073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079054    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container"
Apr 24 22:43:14.079073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079060    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container"
Apr 24 22:43:14.079073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079068    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent"
Apr 24 22:43:14.079073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079072    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent"
Apr 24 22:43:14.079456 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079114    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kserve-container"
Apr 24 22:43:14.079456 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079126    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="agent"
Apr 24 22:43:14.079456 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.079135    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="795d7a1d-84da-48b7-a711-467bca095ba5" containerName="kube-rbac-proxy"
Apr 24 22:43:14.082113 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.082095    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.084722 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.084698    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Apr 24 22:43:14.084810 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.084727    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Apr 24 22:43:14.093297 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.093242    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"]
Apr 24 22:43:14.116784 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.116759    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"]
Apr 24 22:43:14.117053 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.117017    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd" podUID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerName="kserve-container" containerID="cri-o://b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca" gracePeriod=30
Apr 24 22:43:14.117117 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.117054    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd" podUID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerName="kube-rbac-proxy" containerID="cri-o://888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13" gracePeriod=30
Apr 24 22:43:14.173141 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.173111    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kj7s\" (UniqueName: \"kubernetes.io/projected/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kube-api-access-7kj7s\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.173230 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.173144    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.173230 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.173163    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f49cfc9c-9c62-4629-a12d-dec88d449bd9-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.173305 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.173237    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f49cfc9c-9c62-4629-a12d-dec88d449bd9-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.273823 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.273789    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7kj7s\" (UniqueName: \"kubernetes.io/projected/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kube-api-access-7kj7s\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.273823 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.273823    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.274024 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.273841    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f49cfc9c-9c62-4629-a12d-dec88d449bd9-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.274024 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.273881    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f49cfc9c-9c62-4629-a12d-dec88d449bd9-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.274247 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.274221    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.274551 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.274492    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f49cfc9c-9c62-4629-a12d-dec88d449bd9-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.276356 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.276322    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f49cfc9c-9c62-4629-a12d-dec88d449bd9-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.282591 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.282567    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kj7s\" (UniqueName: \"kubernetes.io/projected/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kube-api-access-7kj7s\") pod \"isvc-lightgbm-predictor-bdf964bd-l5j69\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.352049 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.352029    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:43:14.393885 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.393862    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:14.464939 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.464910    2543 generic.go:358] "Generic (PLEG): container finished" podID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerID="cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a" exitCode=2
Apr 24 22:43:14.465086 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.464986    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerDied","Data":"cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a"}
Apr 24 22:43:14.466416 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.466395    2543 generic.go:358] "Generic (PLEG): container finished" podID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerID="888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13" exitCode=2
Apr 24 22:43:14.466416 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.466412    2543 generic.go:358] "Generic (PLEG): container finished" podID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerID="b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca" exitCode=2
Apr 24 22:43:14.466560 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.466464    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd" event={"ID":"d81e7f17-a9f6-4c1a-bbef-23886621e642","Type":"ContainerDied","Data":"888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13"}
Apr 24 22:43:14.466560 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.466473    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"
Apr 24 22:43:14.466560 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.466488    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd" event={"ID":"d81e7f17-a9f6-4c1a-bbef-23886621e642","Type":"ContainerDied","Data":"b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca"}
Apr 24 22:43:14.466560 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.466498    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd" event={"ID":"d81e7f17-a9f6-4c1a-bbef-23886621e642","Type":"ContainerDied","Data":"3a42351585c7f3420ae410936161099281230aef270313f157c9d8d79a242816"}
Apr 24 22:43:14.466560 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.466511    2543 scope.go:117] "RemoveContainer" containerID="888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13"
Apr 24 22:43:14.474869 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.474849    2543 scope.go:117] "RemoveContainer" containerID="b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca"
Apr 24 22:43:14.475114 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.475041    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81e7f17-a9f6-4c1a-bbef-23886621e642-message-dumper-kube-rbac-proxy-sar-config\") pod \"d81e7f17-a9f6-4c1a-bbef-23886621e642\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") "
Apr 24 22:43:14.475114 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.475081    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn9zt\" (UniqueName: \"kubernetes.io/projected/d81e7f17-a9f6-4c1a-bbef-23886621e642-kube-api-access-qn9zt\") pod \"d81e7f17-a9f6-4c1a-bbef-23886621e642\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") "
Apr 24 22:43:14.475251 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.475133    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81e7f17-a9f6-4c1a-bbef-23886621e642-proxy-tls\") pod \"d81e7f17-a9f6-4c1a-bbef-23886621e642\" (UID: \"d81e7f17-a9f6-4c1a-bbef-23886621e642\") "
Apr 24 22:43:14.475445 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.475406    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81e7f17-a9f6-4c1a-bbef-23886621e642-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "d81e7f17-a9f6-4c1a-bbef-23886621e642" (UID: "d81e7f17-a9f6-4c1a-bbef-23886621e642"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:43:14.475541 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.475520    2543 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d81e7f17-a9f6-4c1a-bbef-23886621e642-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:43:14.478519 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.478494    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81e7f17-a9f6-4c1a-bbef-23886621e642-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d81e7f17-a9f6-4c1a-bbef-23886621e642" (UID: "d81e7f17-a9f6-4c1a-bbef-23886621e642"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:43:14.480663 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.480640    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81e7f17-a9f6-4c1a-bbef-23886621e642-kube-api-access-qn9zt" (OuterVolumeSpecName: "kube-api-access-qn9zt") pod "d81e7f17-a9f6-4c1a-bbef-23886621e642" (UID: "d81e7f17-a9f6-4c1a-bbef-23886621e642"). InnerVolumeSpecName "kube-api-access-qn9zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:43:14.483933 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.483800    2543 scope.go:117] "RemoveContainer" containerID="888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13"
Apr 24 22:43:14.484081 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:43:14.484058    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13\": container with ID starting with 888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13 not found: ID does not exist" containerID="888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13"
Apr 24 22:43:14.484150 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.484087    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13"} err="failed to get container status \"888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13\": rpc error: code = NotFound desc = could not find container \"888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13\": container with ID starting with 888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13 not found: ID does not exist"
Apr 24 22:43:14.484150 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.484104    2543 scope.go:117] "RemoveContainer" containerID="b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca"
Apr 24 22:43:14.484319 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:43:14.484302    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca\": container with ID starting with b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca not found: ID does not exist" containerID="b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca"
Apr 24 22:43:14.484421 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.484321    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca"} err="failed to get container status \"b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca\": rpc error: code = NotFound desc = could not find container \"b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca\": container with ID starting with b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca not found: ID does not exist"
Apr 24 22:43:14.484421 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.484353    2543 scope.go:117] "RemoveContainer" containerID="888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13"
Apr 24 22:43:14.484589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.484570    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13"} err="failed to get container status \"888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13\": rpc error: code = NotFound desc = could not find container \"888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13\": container with ID starting with 888e91c3540219e52a8a74b1c0ebb270e570c2c4ac6d9c27a0550b9489073a13 not found: ID does not exist"
Apr 24 22:43:14.484589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.484587    2543 scope.go:117] "RemoveContainer" containerID="b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca"
Apr 24 22:43:14.484836 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.484813    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca"} err="failed to get container status \"b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca\": rpc error: code = NotFound desc = could not find container \"b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca\": container with ID starting with b4f87db4abd4f7d73d10695cdccd3012a23751df0caf28df9639bfec79ac3fca not found: ID does not exist"
Apr 24 22:43:14.516047 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.515946    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"]
Apr 24 22:43:14.518671 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:43:14.518649    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49cfc9c_9c62_4629_a12d_dec88d449bd9.slice/crio-4d8ff6573bda87d369f0a1164cc0810bdd64658b55e7034fa665b7d1135229cb WatchSource:0}: Error finding container 4d8ff6573bda87d369f0a1164cc0810bdd64658b55e7034fa665b7d1135229cb: Status 404 returned error can't find the container with id 4d8ff6573bda87d369f0a1164cc0810bdd64658b55e7034fa665b7d1135229cb
Apr 24 22:43:14.576194 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.576176    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d81e7f17-a9f6-4c1a-bbef-23886621e642-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:43:14.576289 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.576197    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn9zt\" (UniqueName: \"kubernetes.io/projected/d81e7f17-a9f6-4c1a-bbef-23886621e642-kube-api-access-qn9zt\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:43:14.787939 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.787912    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"]
Apr 24 22:43:14.790623 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:14.790601    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-p9nkd"]
Apr 24 22:43:15.055543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:15.055513    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d81e7f17-a9f6-4c1a-bbef-23886621e642" path="/var/lib/kubelet/pods/d81e7f17-a9f6-4c1a-bbef-23886621e642/volumes"
Apr 24 22:43:15.470115 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:15.470071    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" event={"ID":"f49cfc9c-9c62-4629-a12d-dec88d449bd9","Type":"ContainerStarted","Data":"85bd67ad01b5b56e0a5393bb70b53c0dc716a61d851275fe02a0a05bc040e1f9"}
Apr 24 22:43:15.470115 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:15.470110    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" event={"ID":"f49cfc9c-9c62-4629-a12d-dec88d449bd9","Type":"ContainerStarted","Data":"4d8ff6573bda87d369f0a1164cc0810bdd64658b55e7034fa665b7d1135229cb"}
Apr 24 22:43:17.238294 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:17.238247    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Apr 24 22:43:18.481699 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:18.481667    2543 generic.go:358] "Generic (PLEG): container finished" podID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerID="95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7" exitCode=0
Apr 24 22:43:18.482021 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:18.481742    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerDied","Data":"95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7"}
Apr 24 22:43:18.482983 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:18.482963    2543 generic.go:358] "Generic (PLEG): container finished" podID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerID="85bd67ad01b5b56e0a5393bb70b53c0dc716a61d851275fe02a0a05bc040e1f9" exitCode=0
Apr 24 22:43:18.483050 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:18.483021    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" event={"ID":"f49cfc9c-9c62-4629-a12d-dec88d449bd9","Type":"ContainerDied","Data":"85bd67ad01b5b56e0a5393bb70b53c0dc716a61d851275fe02a0a05bc040e1f9"}
Apr 24 22:43:22.238007 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:22.237958    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Apr 24 22:43:22.242461 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:22.242425    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:43:22.242805 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:22.242781    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:43:25.508641 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:25.508607    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" event={"ID":"f49cfc9c-9c62-4629-a12d-dec88d449bd9","Type":"ContainerStarted","Data":"0632b225e69d28f9e3fe02591be863cd06f97c774cb18033ce45afe85e0b629b"}
Apr 24 22:43:25.508641 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:25.508647    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" event={"ID":"f49cfc9c-9c62-4629-a12d-dec88d449bd9","Type":"ContainerStarted","Data":"071d009bec47a4b8decdad56d94b6e1cfebdf89d616b264e679fad91bda1f78d"}
Apr 24 22:43:25.509090 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:25.508845    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:25.528442 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:25.528400    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podStartSLOduration=5.252656581 podStartE2EDuration="11.528387424s" podCreationTimestamp="2026-04-24 22:43:14 +0000 UTC" firstStartedPulling="2026-04-24 22:43:18.48420937 +0000 UTC m=+801.910606878" lastFinishedPulling="2026-04-24 22:43:24.75994021 +0000 UTC m=+808.186337721" observedRunningTime="2026-04-24 22:43:25.527599569 +0000 UTC m=+808.953997100" watchObservedRunningTime="2026-04-24 22:43:25.528387424 +0000 UTC m=+808.954785010"
Apr 24 22:43:26.512429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:26.512403    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:26.513272 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:26.513247    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Apr 24 22:43:27.237630 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:27.237591    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Apr 24 22:43:27.237815 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:27.237711    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:43:27.515618 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:27.515535    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Apr 24 22:43:32.238419 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:32.238375    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Apr 24 22:43:32.242746 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:32.242701    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:43:32.243119 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:32.243096    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:43:32.519827 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:32.519754    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:43:32.520409 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:32.520384    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Apr 24 22:43:37.237602 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:37.237564    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Apr 24 22:43:42.237801 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:42.237765    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Apr 24 22:43:42.242583 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:42.242553    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Apr 24 22:43:42.242712 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:42.242652    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:43:42.242891 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:42.242869    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 24 22:43:42.242983 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:42.242969    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:43:42.520539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:42.520471    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Apr 24 22:43:44.161321 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.161300    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:43:44.297027 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.296960    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-proxy-tls\") pod \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") "
Apr 24 22:43:44.297163 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.297026    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kserve-provision-location\") pod \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") "
Apr 24 22:43:44.297163 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.297057    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-isvc-logger-kube-rbac-proxy-sar-config\") pod \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") "
Apr 24 22:43:44.297163 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.297084    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhlh9\" (UniqueName: \"kubernetes.io/projected/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kube-api-access-hhlh9\") pod \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\" (UID: \"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb\") "
Apr 24 22:43:44.297418 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.297382    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" (UID: "2d2bd2bc-d9d8-4210-82c8-5e215265e1eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:43:44.297532 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.297424    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" (UID: "2d2bd2bc-d9d8-4210-82c8-5e215265e1eb"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:43:44.299146 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.299120    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kube-api-access-hhlh9" (OuterVolumeSpecName: "kube-api-access-hhlh9") pod "2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" (UID: "2d2bd2bc-d9d8-4210-82c8-5e215265e1eb"). InnerVolumeSpecName "kube-api-access-hhlh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:43:44.299146 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.299130    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" (UID: "2d2bd2bc-d9d8-4210-82c8-5e215265e1eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:43:44.397841 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.397810    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:43:44.397841 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.397841    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:43:44.397841 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.397852    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhlh9\" (UniqueName: \"kubernetes.io/projected/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-kube-api-access-hhlh9\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:43:44.398006 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.397862    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:43:44.563064 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.563005    2543 generic.go:358] "Generic (PLEG): container finished" podID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerID="bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383" exitCode=0
Apr 24 22:43:44.563152 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.563072    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerDied","Data":"bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383"}
Apr 24 22:43:44.563152 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.563099    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd" event={"ID":"2d2bd2bc-d9d8-4210-82c8-5e215265e1eb","Type":"ContainerDied","Data":"7d14a94ab95b52ac4afdbee85eb680ea901d27e265382b21160515fa702682ce"}
Apr 24 22:43:44.563152 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.563102    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"
Apr 24 22:43:44.563152 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.563113    2543 scope.go:117] "RemoveContainer" containerID="bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383"
Apr 24 22:43:44.573238 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.573214    2543 scope.go:117] "RemoveContainer" containerID="cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a"
Apr 24 22:43:44.580372 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.580332    2543 scope.go:117] "RemoveContainer" containerID="95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7"
Apr 24 22:43:44.588591 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.588574    2543 scope.go:117] "RemoveContainer" containerID="9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb"
Apr 24 22:43:44.593277 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.593258    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"]
Apr 24 22:43:44.596629 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.596609    2543 scope.go:117] "RemoveContainer" containerID="bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383"
Apr 24 22:43:44.596896 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:43:44.596877    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383\": container with ID starting with bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383 not found: ID does not exist" containerID="bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383"
Apr 24 22:43:44.596966 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.596907    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383"} err="failed to get container status \"bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383\": rpc error: code = NotFound desc = could not find container \"bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383\": container with ID starting with bfb3239e44a4c5f9900c0eef94b23b3a3be32a1d570f8feb14782aba412c6383 not found: ID does not exist"
Apr 24 22:43:44.596966 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.596926    2543 scope.go:117] "RemoveContainer" containerID="cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a"
Apr 24 22:43:44.597148 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:43:44.597129    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a\": container with ID starting with cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a not found: ID does not exist" containerID="cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a"
Apr 24 22:43:44.597209 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.597153    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a"} err="failed to get container status \"cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a\": rpc error: code = NotFound desc = could not find container \"cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a\": container with ID starting with cb694c82e0d3e55738e96a388c8144161ca7545b2c3de012b0d6bdb648c0170a not found: ID does not exist"
Apr 24 22:43:44.597209 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.597169    2543 scope.go:117] "RemoveContainer" containerID="95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7"
Apr 24 22:43:44.597425 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:43:44.597396    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7\": container with ID starting with 95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7 not found: ID does not exist" containerID="95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7"
Apr 24 22:43:44.597425 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.597417    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7"} err="failed to get container status \"95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7\": rpc error: code = NotFound desc = could not find container \"95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7\": container with ID starting with 95fd59c6d8a50da862f69d168c05728651d41650f5abfc4879e5359a9997e6e7 not found: ID does not exist"
Apr 24 22:43:44.597533 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.597433    2543 scope.go:117] "RemoveContainer" containerID="9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb"
Apr 24 22:43:44.597691 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:43:44.597667    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb\": container with ID starting with 9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb not found: ID does not exist" containerID="9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb"
Apr 24 22:43:44.597755 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.597697    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb"} err="failed to get container status \"9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb\": rpc error: code = NotFound desc = could not find container \"9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb\": container with ID starting with 9e2a1478e21bc7889642f485891d2080e8cbd958274c5e6df2e8c4a8f848b4cb not found: ID does not exist"
Apr 24 22:43:44.598248 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:44.598227    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7b48948c-mjqrd"]
Apr 24 22:43:45.055679 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:45.055642    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" path="/var/lib/kubelet/pods/2d2bd2bc-d9d8-4210-82c8-5e215265e1eb/volumes"
Apr 24 22:43:52.521305 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:43:52.521260    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Apr 24 22:44:02.520408 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:02.520367    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Apr 24 22:44:12.520568 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:12.520524    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Apr 24 22:44:22.520400 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:22.520360    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Apr 24 22:44:32.520330 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:32.520287    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Apr 24 22:44:42.521684 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:42.521610    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:44:44.194150 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.194120    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"]
Apr 24 22:44:44.194566 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.194471    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container" containerID="cri-o://071d009bec47a4b8decdad56d94b6e1cfebdf89d616b264e679fad91bda1f78d" gracePeriod=30
Apr 24 22:44:44.194566 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.194491    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kube-rbac-proxy" containerID="cri-o://0632b225e69d28f9e3fe02591be863cd06f97c774cb18033ce45afe85e0b629b" gracePeriod=30
Apr 24 22:44:44.339778 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.339744    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"]
Apr 24 22:44:44.340040 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340027    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerName="kserve-container"
Apr 24 22:44:44.340040 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340040    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerName="kserve-container"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340055    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340061    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340071    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340076    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340088    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerName="kube-rbac-proxy"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340093    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerName="kube-rbac-proxy"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340100    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340105    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340113    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="storage-initializer"
Apr 24 22:44:44.340131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340118    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="storage-initializer"
Apr 24 22:44:44.340468 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340160    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="agent"
Apr 24 22:44:44.340468 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340171    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kube-rbac-proxy"
Apr 24 22:44:44.340468 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340179    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerName="kserve-container"
Apr 24 22:44:44.340468 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340184    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="d81e7f17-a9f6-4c1a-bbef-23886621e642" containerName="kube-rbac-proxy"
Apr 24 22:44:44.340468 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.340190    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2d2bd2bc-d9d8-4210-82c8-5e215265e1eb" containerName="kserve-container"
Apr 24 22:44:44.347030 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.347005    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.349675 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.349653    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Apr 24 22:44:44.349864 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.349848    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Apr 24 22:44:44.354405 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.354387    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"]
Apr 24 22:44:44.436983 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.436960    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d646a398-b0bd-4052-8ca3-70bf26e2be30-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.437119 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.437003    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d646a398-b0bd-4052-8ca3-70bf26e2be30-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.437119 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.437044    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d646a398-b0bd-4052-8ca3-70bf26e2be30-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.437119 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.437085    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pc96\" (UniqueName: \"kubernetes.io/projected/d646a398-b0bd-4052-8ca3-70bf26e2be30-kube-api-access-5pc96\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.537698 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.537673    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d646a398-b0bd-4052-8ca3-70bf26e2be30-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.537805 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.537719    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d646a398-b0bd-4052-8ca3-70bf26e2be30-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.537805 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.537777    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d646a398-b0bd-4052-8ca3-70bf26e2be30-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.537900 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.537805    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5pc96\" (UniqueName: \"kubernetes.io/projected/d646a398-b0bd-4052-8ca3-70bf26e2be30-kube-api-access-5pc96\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.538154 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.538135    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d646a398-b0bd-4052-8ca3-70bf26e2be30-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.538381 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.538328    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d646a398-b0bd-4052-8ca3-70bf26e2be30-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.540178 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.540157    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d646a398-b0bd-4052-8ca3-70bf26e2be30-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.547719 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.547700    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pc96\" (UniqueName: \"kubernetes.io/projected/d646a398-b0bd-4052-8ca3-70bf26e2be30-kube-api-access-5pc96\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.657963 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.657939    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:44.745052 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.745014    2543 generic.go:358] "Generic (PLEG): container finished" podID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerID="0632b225e69d28f9e3fe02591be863cd06f97c774cb18033ce45afe85e0b629b" exitCode=2
Apr 24 22:44:44.745168 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.745076    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" event={"ID":"f49cfc9c-9c62-4629-a12d-dec88d449bd9","Type":"ContainerDied","Data":"0632b225e69d28f9e3fe02591be863cd06f97c774cb18033ce45afe85e0b629b"}
Apr 24 22:44:44.779126 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:44.779107    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"]
Apr 24 22:44:44.781315 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:44:44.781293    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd646a398_b0bd_4052_8ca3_70bf26e2be30.slice/crio-abd9fce7338ea32b60b3c94bd755f7cbccc87f69ef447ce673001abdc3e72a71 WatchSource:0}: Error finding container abd9fce7338ea32b60b3c94bd755f7cbccc87f69ef447ce673001abdc3e72a71: Status 404 returned error can't find the container with id abd9fce7338ea32b60b3c94bd755f7cbccc87f69ef447ce673001abdc3e72a71
Apr 24 22:44:45.749300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:45.749267    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" event={"ID":"d646a398-b0bd-4052-8ca3-70bf26e2be30","Type":"ContainerStarted","Data":"62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116"}
Apr 24 22:44:45.749658 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:45.749316    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" event={"ID":"d646a398-b0bd-4052-8ca3-70bf26e2be30","Type":"ContainerStarted","Data":"abd9fce7338ea32b60b3c94bd755f7cbccc87f69ef447ce673001abdc3e72a71"}
Apr 24 22:44:47.516376 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:47.516309    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Apr 24 22:44:48.760117 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.760089    2543 generic.go:358] "Generic (PLEG): container finished" podID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerID="071d009bec47a4b8decdad56d94b6e1cfebdf89d616b264e679fad91bda1f78d" exitCode=0
Apr 24 22:44:48.760515 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.760165    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" event={"ID":"f49cfc9c-9c62-4629-a12d-dec88d449bd9","Type":"ContainerDied","Data":"071d009bec47a4b8decdad56d94b6e1cfebdf89d616b264e679fad91bda1f78d"}
Apr 24 22:44:48.764485 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.764465    2543 generic.go:358] "Generic (PLEG): container finished" podID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerID="62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116" exitCode=0
Apr 24 22:44:48.764569 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.764516    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" event={"ID":"d646a398-b0bd-4052-8ca3-70bf26e2be30","Type":"ContainerDied","Data":"62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116"}
Apr 24 22:44:48.839207 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.839188    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:44:48.974085 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.974056    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f49cfc9c-9c62-4629-a12d-dec88d449bd9-proxy-tls\") pod \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") "
Apr 24 22:44:48.974250 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.974099    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f49cfc9c-9c62-4629-a12d-dec88d449bd9-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") "
Apr 24 22:44:48.974250 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.974127    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kserve-provision-location\") pod \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") "
Apr 24 22:44:48.974250 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.974206    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kj7s\" (UniqueName: \"kubernetes.io/projected/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kube-api-access-7kj7s\") pod \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\" (UID: \"f49cfc9c-9c62-4629-a12d-dec88d449bd9\") "
Apr 24 22:44:48.974454 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.974418    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49cfc9c-9c62-4629-a12d-dec88d449bd9-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "f49cfc9c-9c62-4629-a12d-dec88d449bd9" (UID: "f49cfc9c-9c62-4629-a12d-dec88d449bd9"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:44:48.974512 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.974495    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f49cfc9c-9c62-4629-a12d-dec88d449bd9" (UID: "f49cfc9c-9c62-4629-a12d-dec88d449bd9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:44:48.976309 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.976284    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kube-api-access-7kj7s" (OuterVolumeSpecName: "kube-api-access-7kj7s") pod "f49cfc9c-9c62-4629-a12d-dec88d449bd9" (UID: "f49cfc9c-9c62-4629-a12d-dec88d449bd9"). InnerVolumeSpecName "kube-api-access-7kj7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:44:48.976445 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:48.976319    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49cfc9c-9c62-4629-a12d-dec88d449bd9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f49cfc9c-9c62-4629-a12d-dec88d449bd9" (UID: "f49cfc9c-9c62-4629-a12d-dec88d449bd9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:44:49.074829 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.074805    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kj7s\" (UniqueName: \"kubernetes.io/projected/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kube-api-access-7kj7s\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:44:49.074829 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.074827    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f49cfc9c-9c62-4629-a12d-dec88d449bd9-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:44:49.074947 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.074837    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f49cfc9c-9c62-4629-a12d-dec88d449bd9-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:44:49.074947 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.074846    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f49cfc9c-9c62-4629-a12d-dec88d449bd9-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:44:49.769402 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.769290    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69" event={"ID":"f49cfc9c-9c62-4629-a12d-dec88d449bd9","Type":"ContainerDied","Data":"4d8ff6573bda87d369f0a1164cc0810bdd64658b55e7034fa665b7d1135229cb"}
Apr 24 22:44:49.769402 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.769363    2543 scope.go:117] "RemoveContainer" containerID="0632b225e69d28f9e3fe02591be863cd06f97c774cb18033ce45afe85e0b629b"
Apr 24 22:44:49.769402 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.769375    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"
Apr 24 22:44:49.783317 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.783287    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" event={"ID":"d646a398-b0bd-4052-8ca3-70bf26e2be30","Type":"ContainerStarted","Data":"e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1"}
Apr 24 22:44:49.783480 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.783324    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" event={"ID":"d646a398-b0bd-4052-8ca3-70bf26e2be30","Type":"ContainerStarted","Data":"fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb"}
Apr 24 22:44:49.783701 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.783678    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:49.786263 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.786247    2543 scope.go:117] "RemoveContainer" containerID="071d009bec47a4b8decdad56d94b6e1cfebdf89d616b264e679fad91bda1f78d"
Apr 24 22:44:49.792647 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.792625    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"]
Apr 24 22:44:49.793792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.793779    2543 scope.go:117] "RemoveContainer" containerID="85bd67ad01b5b56e0a5393bb70b53c0dc716a61d851275fe02a0a05bc040e1f9"
Apr 24 22:44:49.798078 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.798055    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-l5j69"]
Apr 24 22:44:49.826464 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:49.826427    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podStartSLOduration=5.826415057 podStartE2EDuration="5.826415057s" podCreationTimestamp="2026-04-24 22:44:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:44:49.825180736 +0000 UTC m=+893.251578265" watchObservedRunningTime="2026-04-24 22:44:49.826415057 +0000 UTC m=+893.252812587"
Apr 24 22:44:50.787804 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:50.787769    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:50.789015 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:50.788987    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:44:51.055767 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:51.055698    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" path="/var/lib/kubelet/pods/f49cfc9c-9c62-4629-a12d-dec88d449bd9/volumes"
Apr 24 22:44:51.790359 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:51.790308    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:44:56.795078 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:56.795050    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:44:56.795532 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:56.795497    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:44:57.029026 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:57.028987    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:44:57.029772 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:44:57.029749    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:45:06.796298 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:45:06.796259    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:45:16.796193 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:45:16.796151    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:45:26.796161 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:45:26.796118    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:45:36.795796 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:45:36.795762    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:45:46.796245 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:45:46.796210    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:45:56.795995 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:45:56.795958    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:46:06.796273 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:06.796192    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:46:14.570962 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.570930    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"]
Apr 24 22:46:14.571449 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.571266    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" containerID="cri-o://fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb" gracePeriod=30
Apr 24 22:46:14.571449 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.571304    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kube-rbac-proxy" containerID="cri-o://e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1" gracePeriod=30
Apr 24 22:46:14.714624 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.714592    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"]
Apr 24 22:46:14.714916 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.714903    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="storage-initializer"
Apr 24 22:46:14.714958 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.714918    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="storage-initializer"
Apr 24 22:46:14.714958 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.714930    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container"
Apr 24 22:46:14.714958 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.714935    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container"
Apr 24 22:46:14.714958 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.714953    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kube-rbac-proxy"
Apr 24 22:46:14.715078 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.714960    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kube-rbac-proxy"
Apr 24 22:46:14.715078 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.715007    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kserve-container"
Apr 24 22:46:14.715078 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.715017    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="f49cfc9c-9c62-4629-a12d-dec88d449bd9" containerName="kube-rbac-proxy"
Apr 24 22:46:14.717906 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.717892    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.720589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.720564    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Apr 24 22:46:14.720698 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.720641    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Apr 24 22:46:14.728548 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.728528    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"]
Apr 24 22:46:14.807323 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.807299    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6337a391-f0c6-41e4-a3f0-32829e227c0d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.807459 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.807328    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6337a391-f0c6-41e4-a3f0-32829e227c0d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.807503 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.807460    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6337a391-f0c6-41e4-a3f0-32829e227c0d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.807503 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.807492    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzc6\" (UniqueName: \"kubernetes.io/projected/6337a391-f0c6-41e4-a3f0-32829e227c0d-kube-api-access-nkzc6\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.909589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.909525    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzc6\" (UniqueName: \"kubernetes.io/projected/6337a391-f0c6-41e4-a3f0-32829e227c0d-kube-api-access-nkzc6\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.909589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.909577    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6337a391-f0c6-41e4-a3f0-32829e227c0d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.909770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.909605    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6337a391-f0c6-41e4-a3f0-32829e227c0d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.909770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.909677    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6337a391-f0c6-41e4-a3f0-32829e227c0d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.910022 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.910003    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6337a391-f0c6-41e4-a3f0-32829e227c0d-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.910294 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.910275    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6337a391-f0c6-41e4-a3f0-32829e227c0d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.912110 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.912093    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6337a391-f0c6-41e4-a3f0-32829e227c0d-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:14.919257 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:14.919232    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzc6\" (UniqueName: \"kubernetes.io/projected/6337a391-f0c6-41e4-a3f0-32829e227c0d-kube-api-access-nkzc6\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:15.028589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:15.028564    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:46:15.043673 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:15.043648    2543 generic.go:358] "Generic (PLEG): container finished" podID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerID="e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1" exitCode=2
Apr 24 22:46:15.043773 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:15.043695    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" event={"ID":"d646a398-b0bd-4052-8ca3-70bf26e2be30","Type":"ContainerDied","Data":"e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1"}
Apr 24 22:46:15.154224 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:15.154200    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"]
Apr 24 22:46:15.156553 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:46:15.156527    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6337a391_f0c6_41e4_a3f0_32829e227c0d.slice/crio-5200103da61cd86b3ab72adab0c732ae6bf71a689452202c354aca1f1bb7d06e WatchSource:0}: Error finding container 5200103da61cd86b3ab72adab0c732ae6bf71a689452202c354aca1f1bb7d06e: Status 404 returned error can't find the container with id 5200103da61cd86b3ab72adab0c732ae6bf71a689452202c354aca1f1bb7d06e
Apr 24 22:46:16.048135 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:16.048097    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" event={"ID":"6337a391-f0c6-41e4-a3f0-32829e227c0d","Type":"ContainerStarted","Data":"9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082"}
Apr 24 22:46:16.048135 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:16.048133    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" event={"ID":"6337a391-f0c6-41e4-a3f0-32829e227c0d","Type":"ContainerStarted","Data":"5200103da61cd86b3ab72adab0c732ae6bf71a689452202c354aca1f1bb7d06e"}
Apr 24 22:46:16.791536 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:16.791487    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Apr 24 22:46:16.795841 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:16.795813    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Apr 24 22:46:19.056679 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.056599    2543 generic.go:358] "Generic (PLEG): container finished" podID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerID="9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082" exitCode=0
Apr 24 22:46:19.056679 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.056636    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" event={"ID":"6337a391-f0c6-41e4-a3f0-32829e227c0d","Type":"ContainerDied","Data":"9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082"}
Apr 24 22:46:19.410907 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.410886    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:46:19.546657 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.546626    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d646a398-b0bd-4052-8ca3-70bf26e2be30-kserve-provision-location\") pod \"d646a398-b0bd-4052-8ca3-70bf26e2be30\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") "
Apr 24 22:46:19.546821 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.546673    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d646a398-b0bd-4052-8ca3-70bf26e2be30-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"d646a398-b0bd-4052-8ca3-70bf26e2be30\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") "
Apr 24 22:46:19.546821 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.546724    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pc96\" (UniqueName: \"kubernetes.io/projected/d646a398-b0bd-4052-8ca3-70bf26e2be30-kube-api-access-5pc96\") pod \"d646a398-b0bd-4052-8ca3-70bf26e2be30\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") "
Apr 24 22:46:19.546946 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.546814    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d646a398-b0bd-4052-8ca3-70bf26e2be30-proxy-tls\") pod \"d646a398-b0bd-4052-8ca3-70bf26e2be30\" (UID: \"d646a398-b0bd-4052-8ca3-70bf26e2be30\") "
Apr 24 22:46:19.547320 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.547144    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d646a398-b0bd-4052-8ca3-70bf26e2be30-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "d646a398-b0bd-4052-8ca3-70bf26e2be30" (UID: "d646a398-b0bd-4052-8ca3-70bf26e2be30"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:46:19.547320 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.547285    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d646a398-b0bd-4052-8ca3-70bf26e2be30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d646a398-b0bd-4052-8ca3-70bf26e2be30" (UID: "d646a398-b0bd-4052-8ca3-70bf26e2be30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:46:19.549546 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.549514    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d646a398-b0bd-4052-8ca3-70bf26e2be30-kube-api-access-5pc96" (OuterVolumeSpecName: "kube-api-access-5pc96") pod "d646a398-b0bd-4052-8ca3-70bf26e2be30" (UID: "d646a398-b0bd-4052-8ca3-70bf26e2be30"). InnerVolumeSpecName "kube-api-access-5pc96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:46:19.550129 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.550091    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d646a398-b0bd-4052-8ca3-70bf26e2be30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d646a398-b0bd-4052-8ca3-70bf26e2be30" (UID: "d646a398-b0bd-4052-8ca3-70bf26e2be30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:46:19.647672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.647603    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pc96\" (UniqueName: \"kubernetes.io/projected/d646a398-b0bd-4052-8ca3-70bf26e2be30-kube-api-access-5pc96\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:46:19.647672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.647639    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d646a398-b0bd-4052-8ca3-70bf26e2be30-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:46:19.647672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.647655    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d646a398-b0bd-4052-8ca3-70bf26e2be30-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:46:19.647672 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:19.647671    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d646a398-b0bd-4052-8ca3-70bf26e2be30-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:46:20.067314 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.067272    2543 generic.go:358] "Generic (PLEG): container finished" podID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerID="fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb" exitCode=0
Apr 24 22:46:20.067793 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.067377    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" event={"ID":"d646a398-b0bd-4052-8ca3-70bf26e2be30","Type":"ContainerDied","Data":"fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb"}
Apr 24 22:46:20.067793 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.067414    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv" event={"ID":"d646a398-b0bd-4052-8ca3-70bf26e2be30","Type":"ContainerDied","Data":"abd9fce7338ea32b60b3c94bd755f7cbccc87f69ef447ce673001abdc3e72a71"}
Apr 24 22:46:20.067793 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.067435    2543 scope.go:117] "RemoveContainer" containerID="e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1"
Apr 24 22:46:20.067793 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.067622    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"
Apr 24 22:46:20.086876 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.086669    2543 scope.go:117] "RemoveContainer" containerID="fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb"
Apr 24 22:46:20.101483 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.101055    2543 scope.go:117] "RemoveContainer" containerID="62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116"
Apr 24 22:46:20.101483 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.101321    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"]
Apr 24 22:46:20.104310 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.104268    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-xg7nv"]
Apr 24 22:46:20.111765 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.111671    2543 scope.go:117] "RemoveContainer" containerID="e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1"
Apr 24 22:46:20.112426 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:46:20.112369    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1\": container with ID starting with e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1 not found: ID does not exist" containerID="e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1"
Apr 24 22:46:20.112426 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.112404    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1"} err="failed to get container status \"e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1\": rpc error: code = NotFound desc = could not find container \"e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1\": container with ID starting with e01204af1e6fe4bd144abebac82ce542b8d067fd2ddd34dbc9bfc7b3cdcbeae1 not found: ID does not exist"
Apr 24 22:46:20.112599 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.112429    2543 scope.go:117] "RemoveContainer" containerID="fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb"
Apr 24 22:46:20.113123 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:46:20.112947    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb\": container with ID starting with fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb not found: ID does not exist" containerID="fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb"
Apr 24 22:46:20.113123 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.112981    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb"} err="failed to get container status \"fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb\": rpc error: code = NotFound desc = could not find container \"fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb\": container with ID starting with fe6f560c6aad8883f1a66de06c01f048af23718472ad6f919895be099d9917bb not found: ID does not exist"
Apr 24 22:46:20.113123 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.113003    2543 scope.go:117] "RemoveContainer" containerID="62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116"
Apr 24 22:46:20.113619 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:46:20.113568    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116\": container with ID starting with 62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116 not found: ID does not exist" containerID="62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116"
Apr 24 22:46:20.113619 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:20.113600    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116"} err="failed to get container status \"62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116\": rpc error: code = NotFound desc = could not find container \"62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116\": container with ID starting with 62ceaf0febdd20f331282d62275a156e1d31685012fd0c85ce82db59b611e116 not found: ID does not exist"
Apr 24 22:46:21.057740 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:46:21.057632    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" path="/var/lib/kubelet/pods/d646a398-b0bd-4052-8ca3-70bf26e2be30/volumes"
Apr 24 22:48:25.784539 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:48:25.784513    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 22:48:26.476958 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:48:26.476919    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" event={"ID":"6337a391-f0c6-41e4-a3f0-32829e227c0d","Type":"ContainerStarted","Data":"1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999"}
Apr 24 22:48:26.476958 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:48:26.476958    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" event={"ID":"6337a391-f0c6-41e4-a3f0-32829e227c0d","Type":"ContainerStarted","Data":"ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace"}
Apr 24 22:48:26.477242 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:48:26.477177    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:48:26.502974 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:48:26.502925    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" podStartSLOduration=5.898759933 podStartE2EDuration="2m12.502913185s" podCreationTimestamp="2026-04-24 22:46:14 +0000 UTC" firstStartedPulling="2026-04-24 22:46:19.057594094 +0000 UTC m=+982.483991602" lastFinishedPulling="2026-04-24 22:48:25.661747343 +0000 UTC m=+1109.088144854" observedRunningTime="2026-04-24 22:48:26.502157401 +0000 UTC m=+1109.928554931" watchObservedRunningTime="2026-04-24 22:48:26.502913185 +0000 UTC m=+1109.929310715"
Apr 24 22:48:27.480274 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:48:27.480241    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:48:33.488114 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:48:33.488088    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:49:03.491474 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:03.491444    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:49:04.805746 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.805718    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"]
Apr 24 22:49:04.806223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.806068    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="kserve-container" containerID="cri-o://ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace" gracePeriod=30
Apr 24 22:49:04.806223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.806115    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="kube-rbac-proxy" containerID="cri-o://1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999" gracePeriod=30
Apr 24 22:49:04.907101 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.907067    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"]
Apr 24 22:49:04.907433 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.907419    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container"
Apr 24 22:49:04.907497 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.907434    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container"
Apr 24 22:49:04.907497 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.907450    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kube-rbac-proxy"
Apr 24 22:49:04.907497 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.907457    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kube-rbac-proxy"
Apr 24 22:49:04.907497 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.907465    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="storage-initializer"
Apr 24 22:49:04.907497 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.907472    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="storage-initializer"
Apr 24 22:49:04.907669 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.907522    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kube-rbac-proxy"
Apr 24 22:49:04.907669 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.907530    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="d646a398-b0bd-4052-8ca3-70bf26e2be30" containerName="kserve-container"
Apr 24 22:49:04.912513 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.912487    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:04.914836 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.914809    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Apr 24 22:49:04.914946 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.914830    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Apr 24 22:49:04.918286 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:04.918267    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"]
Apr 24 22:49:05.004769 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.004733    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/220f48d7-ab53-4f34-9185-86130d94b9ed-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.004769 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.004771    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmrd\" (UniqueName: \"kubernetes.io/projected/220f48d7-ab53-4f34-9185-86130d94b9ed-kube-api-access-4xmrd\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.004997 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.004806    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/220f48d7-ab53-4f34-9185-86130d94b9ed-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.004997 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.004920    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/220f48d7-ab53-4f34-9185-86130d94b9ed-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.106212 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.106143    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/220f48d7-ab53-4f34-9185-86130d94b9ed-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.106212 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.106193    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/220f48d7-ab53-4f34-9185-86130d94b9ed-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.106484 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.106249    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/220f48d7-ab53-4f34-9185-86130d94b9ed-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.106484 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.106275    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmrd\" (UniqueName: \"kubernetes.io/projected/220f48d7-ab53-4f34-9185-86130d94b9ed-kube-api-access-4xmrd\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.106614 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.106591    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/220f48d7-ab53-4f34-9185-86130d94b9ed-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.106972 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.106949    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/220f48d7-ab53-4f34-9185-86130d94b9ed-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.108953 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.108929    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/220f48d7-ab53-4f34-9185-86130d94b9ed-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.113816 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.113785    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmrd\" (UniqueName: \"kubernetes.io/projected/220f48d7-ab53-4f34-9185-86130d94b9ed-kube-api-access-4xmrd\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.225473 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.225445    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:05.349214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.349185    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"]
Apr 24 22:49:05.352397 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:49:05.352371    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220f48d7_ab53_4f34_9185_86130d94b9ed.slice/crio-1bf442d1e9b80d0b314cfaacc077bc733fb9bd6838409a948194b036761904ad WatchSource:0}: Error finding container 1bf442d1e9b80d0b314cfaacc077bc733fb9bd6838409a948194b036761904ad: Status 404 returned error can't find the container with id 1bf442d1e9b80d0b314cfaacc077bc733fb9bd6838409a948194b036761904ad
Apr 24 22:49:05.591027 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.590990    2543 generic.go:358] "Generic (PLEG): container finished" podID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerID="1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999" exitCode=2
Apr 24 22:49:05.591194 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.591056    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" event={"ID":"6337a391-f0c6-41e4-a3f0-32829e227c0d","Type":"ContainerDied","Data":"1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999"}
Apr 24 22:49:05.592519 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.592498    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" event={"ID":"220f48d7-ab53-4f34-9185-86130d94b9ed","Type":"ContainerStarted","Data":"8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1"}
Apr 24 22:49:05.592620 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.592525    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" event={"ID":"220f48d7-ab53-4f34-9185-86130d94b9ed","Type":"ContainerStarted","Data":"1bf442d1e9b80d0b314cfaacc077bc733fb9bd6838409a948194b036761904ad"}
Apr 24 22:49:05.741017 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.740993    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:49:05.812286 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.812255    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6337a391-f0c6-41e4-a3f0-32829e227c0d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"6337a391-f0c6-41e4-a3f0-32829e227c0d\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") "
Apr 24 22:49:05.812715 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.812325    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6337a391-f0c6-41e4-a3f0-32829e227c0d-kserve-provision-location\") pod \"6337a391-f0c6-41e4-a3f0-32829e227c0d\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") "
Apr 24 22:49:05.812715 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.812436    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6337a391-f0c6-41e4-a3f0-32829e227c0d-proxy-tls\") pod \"6337a391-f0c6-41e4-a3f0-32829e227c0d\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") "
Apr 24 22:49:05.812715 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.812464    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkzc6\" (UniqueName: \"kubernetes.io/projected/6337a391-f0c6-41e4-a3f0-32829e227c0d-kube-api-access-nkzc6\") pod \"6337a391-f0c6-41e4-a3f0-32829e227c0d\" (UID: \"6337a391-f0c6-41e4-a3f0-32829e227c0d\") "
Apr 24 22:49:05.812715 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.812596    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6337a391-f0c6-41e4-a3f0-32829e227c0d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "6337a391-f0c6-41e4-a3f0-32829e227c0d" (UID: "6337a391-f0c6-41e4-a3f0-32829e227c0d"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:49:05.812715 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.812665    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337a391-f0c6-41e4-a3f0-32829e227c0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6337a391-f0c6-41e4-a3f0-32829e227c0d" (UID: "6337a391-f0c6-41e4-a3f0-32829e227c0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:49:05.814618 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.814587    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6337a391-f0c6-41e4-a3f0-32829e227c0d-kube-api-access-nkzc6" (OuterVolumeSpecName: "kube-api-access-nkzc6") pod "6337a391-f0c6-41e4-a3f0-32829e227c0d" (UID: "6337a391-f0c6-41e4-a3f0-32829e227c0d"). InnerVolumeSpecName "kube-api-access-nkzc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:49:05.814801 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.814629    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6337a391-f0c6-41e4-a3f0-32829e227c0d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6337a391-f0c6-41e4-a3f0-32829e227c0d" (UID: "6337a391-f0c6-41e4-a3f0-32829e227c0d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:49:05.913223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.913153    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6337a391-f0c6-41e4-a3f0-32829e227c0d-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:49:05.913223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.913177    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6337a391-f0c6-41e4-a3f0-32829e227c0d-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:49:05.913223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.913187    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6337a391-f0c6-41e4-a3f0-32829e227c0d-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:49:05.913223 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:05.913196    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkzc6\" (UniqueName: \"kubernetes.io/projected/6337a391-f0c6-41e4-a3f0-32829e227c0d-kube-api-access-nkzc6\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:49:06.597875 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.597842    2543 generic.go:358] "Generic (PLEG): container finished" podID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerID="ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace" exitCode=0
Apr 24 22:49:06.598077 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.597923    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"
Apr 24 22:49:06.598077 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.597929    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" event={"ID":"6337a391-f0c6-41e4-a3f0-32829e227c0d","Type":"ContainerDied","Data":"ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace"}
Apr 24 22:49:06.598077 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.597967    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd" event={"ID":"6337a391-f0c6-41e4-a3f0-32829e227c0d","Type":"ContainerDied","Data":"5200103da61cd86b3ab72adab0c732ae6bf71a689452202c354aca1f1bb7d06e"}
Apr 24 22:49:06.598077 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.597983    2543 scope.go:117] "RemoveContainer" containerID="1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999"
Apr 24 22:49:06.606283 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.606262    2543 scope.go:117] "RemoveContainer" containerID="ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace"
Apr 24 22:49:06.613187 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.613173    2543 scope.go:117] "RemoveContainer" containerID="9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082"
Apr 24 22:49:06.618913 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.618890    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"]
Apr 24 22:49:06.621718 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.621697    2543 scope.go:117] "RemoveContainer" containerID="1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999"
Apr 24 22:49:06.621994 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:49:06.621976    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999\": container with ID starting with 1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999 not found: ID does not exist" containerID="1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999"
Apr 24 22:49:06.622071 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.622004    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999"} err="failed to get container status \"1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999\": rpc error: code = NotFound desc = could not find container \"1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999\": container with ID starting with 1cdebf3bc346da40d1f1595a7c1568b6d3b4cfca30d73c67fa58081c466bf999 not found: ID does not exist"
Apr 24 22:49:06.622071 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.622028    2543 scope.go:117] "RemoveContainer" containerID="ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace"
Apr 24 22:49:06.622296 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:49:06.622278    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace\": container with ID starting with ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace not found: ID does not exist" containerID="ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace"
Apr 24 22:49:06.622375 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.622303    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace"} err="failed to get container status \"ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace\": rpc error: code = NotFound desc = could not find container \"ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace\": container with ID starting with ab5e71b16ecb45223283e96fcdcf267feb15bb572bb06aefc9043780a1536ace not found: ID does not exist"
Apr 24 22:49:06.622375 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.622350    2543 scope.go:117] "RemoveContainer" containerID="9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082"
Apr 24 22:49:06.622589 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.622571    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-lcnsd"]
Apr 24 22:49:06.622636 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:49:06.622573    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082\": container with ID starting with 9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082 not found: ID does not exist" containerID="9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082"
Apr 24 22:49:06.622636 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:06.622618    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082"} err="failed to get container status \"9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082\": rpc error: code = NotFound desc = could not find container \"9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082\": container with ID starting with 9efed8290383babe7a40469113ff9c2675ed525808f0a54e870a8a4046437082 not found: ID does not exist"
Apr 24 22:49:07.055150 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:07.055124    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" path="/var/lib/kubelet/pods/6337a391-f0c6-41e4-a3f0-32829e227c0d/volumes"
Apr 24 22:49:09.608303 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:09.608238    2543 generic.go:358] "Generic (PLEG): container finished" podID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerID="8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1" exitCode=0
Apr 24 22:49:09.608634 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:09.608310    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" event={"ID":"220f48d7-ab53-4f34-9185-86130d94b9ed","Type":"ContainerDied","Data":"8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1"}
Apr 24 22:49:10.613148 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:10.613111    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" event={"ID":"220f48d7-ab53-4f34-9185-86130d94b9ed","Type":"ContainerStarted","Data":"3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694"}
Apr 24 22:49:10.613148 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:10.613155    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" event={"ID":"220f48d7-ab53-4f34-9185-86130d94b9ed","Type":"ContainerStarted","Data":"c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5"}
Apr 24 22:49:10.613713 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:10.613482    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:10.631708 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:10.631669    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" podStartSLOduration=6.63165503 podStartE2EDuration="6.63165503s" podCreationTimestamp="2026-04-24 22:49:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:49:10.630454367 +0000 UTC m=+1154.056851910" watchObservedRunningTime="2026-04-24 22:49:10.63165503 +0000 UTC m=+1154.058052560"
Apr 24 22:49:11.616364 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:11.616323    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:11.617453 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:11.617423    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Apr 24 22:49:12.618839 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:12.618800    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Apr 24 22:49:17.622902 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:17.622872    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:17.623868 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:17.623849    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:24.934130 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:24.934099    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"]
Apr 24 22:49:24.934621 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:24.934531    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kserve-container" containerID="cri-o://c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5" gracePeriod=30
Apr 24 22:49:24.934621 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:24.934569    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kube-rbac-proxy" containerID="cri-o://3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694" gracePeriod=30
Apr 24 22:49:25.023814 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.023784    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"]
Apr 24 22:49:25.024090 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.024078    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="storage-initializer"
Apr 24 22:49:25.024128 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.024100    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="storage-initializer"
Apr 24 22:49:25.024128 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.024111    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="kserve-container"
Apr 24 22:49:25.024128 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.024117    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="kserve-container"
Apr 24 22:49:25.024227 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.024129    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="kube-rbac-proxy"
Apr 24 22:49:25.024227 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.024135    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="kube-rbac-proxy"
Apr 24 22:49:25.024227 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.024182    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="kube-rbac-proxy"
Apr 24 22:49:25.024227 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.024194    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="6337a391-f0c6-41e4-a3f0-32829e227c0d" containerName="kserve-container"
Apr 24 22:49:25.027132 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.027117    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.029441 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.029417    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Apr 24 22:49:25.029441 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.029427    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Apr 24 22:49:25.034858 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.034836    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"]
Apr 24 22:49:25.156404 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.156356    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnldr\" (UniqueName: \"kubernetes.io/projected/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kube-api-access-jnldr\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.156581 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.156457    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31e4385-23f1-4cd1-8cb1-1649a298d44c-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.156581 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.156496    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31e4385-23f1-4cd1-8cb1-1649a298d44c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.156581 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.156569    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.257956 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.257910    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jnldr\" (UniqueName: \"kubernetes.io/projected/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kube-api-access-jnldr\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.258134 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.257982    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31e4385-23f1-4cd1-8cb1-1649a298d44c-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.258134 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.258018    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31e4385-23f1-4cd1-8cb1-1649a298d44c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.258134 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.258074    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.258285 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:49:25.258180    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Apr 24 22:49:25.258285 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:49:25.258258    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a31e4385-23f1-4cd1-8cb1-1649a298d44c-proxy-tls podName:a31e4385-23f1-4cd1-8cb1-1649a298d44c nodeName:}" failed. No retries permitted until 2026-04-24 22:49:25.758237699 +0000 UTC m=+1169.184635221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a31e4385-23f1-4cd1-8cb1-1649a298d44c-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" (UID: "a31e4385-23f1-4cd1-8cb1-1649a298d44c") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Apr 24 22:49:25.258573 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.258552    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.258766 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.258748    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31e4385-23f1-4cd1-8cb1-1649a298d44c-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.266633 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.266608    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnldr\" (UniqueName: \"kubernetes.io/projected/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kube-api-access-jnldr\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.568034 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.568014    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:25.661058 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.661030    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xmrd\" (UniqueName: \"kubernetes.io/projected/220f48d7-ab53-4f34-9185-86130d94b9ed-kube-api-access-4xmrd\") pod \"220f48d7-ab53-4f34-9185-86130d94b9ed\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") "
Apr 24 22:49:25.661165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.661078    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/220f48d7-ab53-4f34-9185-86130d94b9ed-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"220f48d7-ab53-4f34-9185-86130d94b9ed\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") "
Apr 24 22:49:25.661165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.661107    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/220f48d7-ab53-4f34-9185-86130d94b9ed-kserve-provision-location\") pod \"220f48d7-ab53-4f34-9185-86130d94b9ed\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") "
Apr 24 22:49:25.661165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.661143    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/220f48d7-ab53-4f34-9185-86130d94b9ed-proxy-tls\") pod \"220f48d7-ab53-4f34-9185-86130d94b9ed\" (UID: \"220f48d7-ab53-4f34-9185-86130d94b9ed\") "
Apr 24 22:49:25.661496 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.661468    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220f48d7-ab53-4f34-9185-86130d94b9ed-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "220f48d7-ab53-4f34-9185-86130d94b9ed" (UID: "220f48d7-ab53-4f34-9185-86130d94b9ed"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:49:25.661569 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.661493    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220f48d7-ab53-4f34-9185-86130d94b9ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "220f48d7-ab53-4f34-9185-86130d94b9ed" (UID: "220f48d7-ab53-4f34-9185-86130d94b9ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:49:25.663235 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.663217    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220f48d7-ab53-4f34-9185-86130d94b9ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "220f48d7-ab53-4f34-9185-86130d94b9ed" (UID: "220f48d7-ab53-4f34-9185-86130d94b9ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:49:25.663303 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.663281    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220f48d7-ab53-4f34-9185-86130d94b9ed-kube-api-access-4xmrd" (OuterVolumeSpecName: "kube-api-access-4xmrd") pod "220f48d7-ab53-4f34-9185-86130d94b9ed" (UID: "220f48d7-ab53-4f34-9185-86130d94b9ed"). InnerVolumeSpecName "kube-api-access-4xmrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:49:25.673178 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.673149    2543 generic.go:358] "Generic (PLEG): container finished" podID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerID="3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694" exitCode=2
Apr 24 22:49:25.673178 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.673174    2543 generic.go:358] "Generic (PLEG): container finished" podID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerID="c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5" exitCode=0
Apr 24 22:49:25.673292 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.673179    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" event={"ID":"220f48d7-ab53-4f34-9185-86130d94b9ed","Type":"ContainerDied","Data":"3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694"}
Apr 24 22:49:25.673292 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.673223    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" event={"ID":"220f48d7-ab53-4f34-9185-86130d94b9ed","Type":"ContainerDied","Data":"c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5"}
Apr 24 22:49:25.673292 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.673238    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf" event={"ID":"220f48d7-ab53-4f34-9185-86130d94b9ed","Type":"ContainerDied","Data":"1bf442d1e9b80d0b314cfaacc077bc733fb9bd6838409a948194b036761904ad"}
Apr 24 22:49:25.673292 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.673239    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"
Apr 24 22:49:25.673468 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.673297    2543 scope.go:117] "RemoveContainer" containerID="3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694"
Apr 24 22:49:25.681116 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.681094    2543 scope.go:117] "RemoveContainer" containerID="c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5"
Apr 24 22:49:25.687869 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.687849    2543 scope.go:117] "RemoveContainer" containerID="8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1"
Apr 24 22:49:25.693688 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.693663    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"]
Apr 24 22:49:25.694931 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.694915    2543 scope.go:117] "RemoveContainer" containerID="3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694"
Apr 24 22:49:25.695186 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:49:25.695167    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694\": container with ID starting with 3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694 not found: ID does not exist" containerID="3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694"
Apr 24 22:49:25.695249 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.695197    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694"} err="failed to get container status \"3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694\": rpc error: code = NotFound desc = could not find container \"3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694\": container with ID starting with 3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694 not found: ID does not exist"
Apr 24 22:49:25.695249 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.695214    2543 scope.go:117] "RemoveContainer" containerID="c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5"
Apr 24 22:49:25.695526 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:49:25.695502    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5\": container with ID starting with c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5 not found: ID does not exist" containerID="c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5"
Apr 24 22:49:25.695623 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.695534    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5"} err="failed to get container status \"c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5\": rpc error: code = NotFound desc = could not find container \"c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5\": container with ID starting with c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5 not found: ID does not exist"
Apr 24 22:49:25.695623 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.695551    2543 scope.go:117] "RemoveContainer" containerID="8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1"
Apr 24 22:49:25.696088 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:49:25.696060    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1\": container with ID starting with 8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1 not found: ID does not exist" containerID="8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1"
Apr 24 22:49:25.696190 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.696096    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1"} err="failed to get container status \"8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1\": rpc error: code = NotFound desc = could not find container \"8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1\": container with ID starting with 8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1 not found: ID does not exist"
Apr 24 22:49:25.696190 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.696113    2543 scope.go:117] "RemoveContainer" containerID="3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694"
Apr 24 22:49:25.696415 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.696397    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694"} err="failed to get container status \"3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694\": rpc error: code = NotFound desc = could not find container \"3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694\": container with ID starting with 3e739d4ac362e781b9ae390167601098bc9d66898200fa1a97f297eb3e49d694 not found: ID does not exist"
Apr 24 22:49:25.696479 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.696417    2543 scope.go:117] "RemoveContainer" containerID="c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5"
Apr 24 22:49:25.696663 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.696645    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5"} err="failed to get container status \"c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5\": rpc error: code = NotFound desc = could not find container \"c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5\": container with ID starting with c55fff1c7b82d1fbcdb14ab243a4dd21d375c7e21cc9eefa53168fc099308dc5 not found: ID does not exist"
Apr 24 22:49:25.696714 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.696665    2543 scope.go:117] "RemoveContainer" containerID="8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1"
Apr 24 22:49:25.696879 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.696860    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1"} err="failed to get container status \"8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1\": rpc error: code = NotFound desc = could not find container \"8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1\": container with ID starting with 8fb20c85bda1ea69b75ad549b82f427063bb9ab3f4280d170d9da7e9edfa6af1 not found: ID does not exist"
Apr 24 22:49:25.698133 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.698116    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-xbstf"]
Apr 24 22:49:25.761794 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.761772    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31e4385-23f1-4cd1-8cb1-1649a298d44c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.761871 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.761852    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/220f48d7-ab53-4f34-9185-86130d94b9ed-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:49:25.761871 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.761863    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/220f48d7-ab53-4f34-9185-86130d94b9ed-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:49:25.761980 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.761873    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/220f48d7-ab53-4f34-9185-86130d94b9ed-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:49:25.761980 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.761881    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmrd\" (UniqueName: \"kubernetes.io/projected/220f48d7-ab53-4f34-9185-86130d94b9ed-kube-api-access-4xmrd\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:49:25.764070 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.764050    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31e4385-23f1-4cd1-8cb1-1649a298d44c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:25.937523 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:25.937465    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:26.057601 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:26.057571    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"]
Apr 24 22:49:26.060761 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:49:26.060730    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31e4385_23f1_4cd1_8cb1_1649a298d44c.slice/crio-7f77330094ffd300bce367fff7074f31602f86ab8387a5f2e6a080c441ead818 WatchSource:0}: Error finding container 7f77330094ffd300bce367fff7074f31602f86ab8387a5f2e6a080c441ead818: Status 404 returned error can't find the container with id 7f77330094ffd300bce367fff7074f31602f86ab8387a5f2e6a080c441ead818
Apr 24 22:49:26.677872 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:26.677832    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" event={"ID":"a31e4385-23f1-4cd1-8cb1-1649a298d44c","Type":"ContainerStarted","Data":"b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db"}
Apr 24 22:49:26.677872 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:26.677874    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" event={"ID":"a31e4385-23f1-4cd1-8cb1-1649a298d44c","Type":"ContainerStarted","Data":"7f77330094ffd300bce367fff7074f31602f86ab8387a5f2e6a080c441ead818"}
Apr 24 22:49:27.057689 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:27.057650    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" path="/var/lib/kubelet/pods/220f48d7-ab53-4f34-9185-86130d94b9ed/volumes"
Apr 24 22:49:30.689936 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:30.689904    2543 generic.go:358] "Generic (PLEG): container finished" podID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerID="b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db" exitCode=0
Apr 24 22:49:30.690276 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:30.689949    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" event={"ID":"a31e4385-23f1-4cd1-8cb1-1649a298d44c","Type":"ContainerDied","Data":"b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db"}
Apr 24 22:49:31.694860 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:31.694818    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" event={"ID":"a31e4385-23f1-4cd1-8cb1-1649a298d44c","Type":"ContainerStarted","Data":"423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956"}
Apr 24 22:49:31.694860 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:31.694865    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" event={"ID":"a31e4385-23f1-4cd1-8cb1-1649a298d44c","Type":"ContainerStarted","Data":"99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b"}
Apr 24 22:49:31.695434 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:31.695057    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:31.714058 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:31.714010    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" podStartSLOduration=6.713996409 podStartE2EDuration="6.713996409s" podCreationTimestamp="2026-04-24 22:49:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:49:31.712454221 +0000 UTC m=+1175.138851751" watchObservedRunningTime="2026-04-24 22:49:31.713996409 +0000 UTC m=+1175.140393938"
Apr 24 22:49:32.698322 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:32.698294    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:38.705990 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:38.705961    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:49:57.051987 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:57.051955    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:49:57.053641 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:49:57.053620    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:50:08.710155 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:08.710127    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:50:15.092767 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.092729    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"]
Apr 24 22:50:15.093219 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.093132    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="kserve-container" containerID="cri-o://99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b" gracePeriod=30
Apr 24 22:50:15.093219 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.093165    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="kube-rbac-proxy" containerID="cri-o://423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956" gracePeriod=30
Apr 24 22:50:15.165751 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.165710    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"]
Apr 24 22:50:15.166072 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.166057    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kserve-container"
Apr 24 22:50:15.166072 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.166073    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kserve-container"
Apr 24 22:50:15.166203 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.166085    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kube-rbac-proxy"
Apr 24 22:50:15.166203 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.166091    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kube-rbac-proxy"
Apr 24 22:50:15.166203 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.166103    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="storage-initializer"
Apr 24 22:50:15.166203 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.166109    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="storage-initializer"
Apr 24 22:50:15.166203 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.166157    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kserve-container"
Apr 24 22:50:15.166203 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.166167    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="220f48d7-ab53-4f34-9185-86130d94b9ed" containerName="kube-rbac-proxy"
Apr 24 22:50:15.169361 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.169333    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.171862 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.171840    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Apr 24 22:50:15.172007 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.171990    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Apr 24 22:50:15.179443 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.179401    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"]
Apr 24 22:50:15.318175 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.318136    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.318175 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.318184    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.318471 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.318205    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnbks\" (UniqueName: \"kubernetes.io/projected/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kube-api-access-rnbks\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.318471 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.318304    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.419758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.419641    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.419758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.419683    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbks\" (UniqueName: \"kubernetes.io/projected/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kube-api-access-rnbks\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.419758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.419726    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.420051 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.419792    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.420165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.420138    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.420437 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.420415    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.422415 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.422390    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.428840 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.428814    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnbks\" (UniqueName: \"kubernetes.io/projected/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kube-api-access-rnbks\") pod \"isvc-sklearn-mcp-predictor-c4b9ff587-62qtx\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.480485 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.480445    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:15.604776 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.604750    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"]
Apr 24 22:50:15.606774 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:50:15.606743    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3fd6b3_d5ba_4417_97ff_777aadcd9556.slice/crio-797c3af81e6193cc05cd6bbf15666b48f4f94e1b1bc13ae188c31fe3c7bd4dcb WatchSource:0}: Error finding container 797c3af81e6193cc05cd6bbf15666b48f4f94e1b1bc13ae188c31fe3c7bd4dcb: Status 404 returned error can't find the container with id 797c3af81e6193cc05cd6bbf15666b48f4f94e1b1bc13ae188c31fe3c7bd4dcb
Apr 24 22:50:15.825043 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.825006    2543 generic.go:358] "Generic (PLEG): container finished" podID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerID="423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956" exitCode=2
Apr 24 22:50:15.825227 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.825074    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" event={"ID":"a31e4385-23f1-4cd1-8cb1-1649a298d44c","Type":"ContainerDied","Data":"423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956"}
Apr 24 22:50:15.826522 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.826496    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerStarted","Data":"c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902"}
Apr 24 22:50:15.826629 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:15.826530    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerStarted","Data":"797c3af81e6193cc05cd6bbf15666b48f4f94e1b1bc13ae188c31fe3c7bd4dcb"}
Apr 24 22:50:16.246686 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.246664    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:50:16.428270 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.428187    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31e4385-23f1-4cd1-8cb1-1649a298d44c-proxy-tls\") pod \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") "
Apr 24 22:50:16.428270 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.428232    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31e4385-23f1-4cd1-8cb1-1649a298d44c-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") "
Apr 24 22:50:16.428270 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.428263    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kserve-provision-location\") pod \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") "
Apr 24 22:50:16.428543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.428304    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnldr\" (UniqueName: \"kubernetes.io/projected/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kube-api-access-jnldr\") pod \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\" (UID: \"a31e4385-23f1-4cd1-8cb1-1649a298d44c\") "
Apr 24 22:50:16.428639 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.428604    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31e4385-23f1-4cd1-8cb1-1649a298d44c-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "a31e4385-23f1-4cd1-8cb1-1649a298d44c" (UID: "a31e4385-23f1-4cd1-8cb1-1649a298d44c"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:50:16.428712 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.428676    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a31e4385-23f1-4cd1-8cb1-1649a298d44c" (UID: "a31e4385-23f1-4cd1-8cb1-1649a298d44c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:50:16.430542 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.430515    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kube-api-access-jnldr" (OuterVolumeSpecName: "kube-api-access-jnldr") pod "a31e4385-23f1-4cd1-8cb1-1649a298d44c" (UID: "a31e4385-23f1-4cd1-8cb1-1649a298d44c"). InnerVolumeSpecName "kube-api-access-jnldr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:50:16.430615 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.430540    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31e4385-23f1-4cd1-8cb1-1649a298d44c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a31e4385-23f1-4cd1-8cb1-1649a298d44c" (UID: "a31e4385-23f1-4cd1-8cb1-1649a298d44c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:50:16.529659 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.529636    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31e4385-23f1-4cd1-8cb1-1649a298d44c-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:50:16.529659 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.529657    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31e4385-23f1-4cd1-8cb1-1649a298d44c-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:50:16.529788 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.529668    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:50:16.529788 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.529679    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnldr\" (UniqueName: \"kubernetes.io/projected/a31e4385-23f1-4cd1-8cb1-1649a298d44c-kube-api-access-jnldr\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:50:16.831153 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.831120    2543 generic.go:358] "Generic (PLEG): container finished" podID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerID="99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b" exitCode=0
Apr 24 22:50:16.831320 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.831210    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" event={"ID":"a31e4385-23f1-4cd1-8cb1-1649a298d44c","Type":"ContainerDied","Data":"99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b"}
Apr 24 22:50:16.831320 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.831232    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"
Apr 24 22:50:16.831320 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.831257    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85" event={"ID":"a31e4385-23f1-4cd1-8cb1-1649a298d44c","Type":"ContainerDied","Data":"7f77330094ffd300bce367fff7074f31602f86ab8387a5f2e6a080c441ead818"}
Apr 24 22:50:16.831320 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.831275    2543 scope.go:117] "RemoveContainer" containerID="423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956"
Apr 24 22:50:16.839319 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.839303    2543 scope.go:117] "RemoveContainer" containerID="99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b"
Apr 24 22:50:16.846236 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.846219    2543 scope.go:117] "RemoveContainer" containerID="b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db"
Apr 24 22:50:16.851757 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.851737    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"]
Apr 24 22:50:16.853241 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.853226    2543 scope.go:117] "RemoveContainer" containerID="423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956"
Apr 24 22:50:16.853492 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:50:16.853473    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956\": container with ID starting with 423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956 not found: ID does not exist" containerID="423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956"
Apr 24 22:50:16.853552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.853499    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956"} err="failed to get container status \"423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956\": rpc error: code = NotFound desc = could not find container \"423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956\": container with ID starting with 423003c979608db8c7505b2641444fdeb5096f447060e30a7570409f68343956 not found: ID does not exist"
Apr 24 22:50:16.853552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.853516    2543 scope.go:117] "RemoveContainer" containerID="99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b"
Apr 24 22:50:16.853744 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:50:16.853730    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b\": container with ID starting with 99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b not found: ID does not exist" containerID="99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b"
Apr 24 22:50:16.853792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.853747    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b"} err="failed to get container status \"99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b\": rpc error: code = NotFound desc = could not find container \"99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b\": container with ID starting with 99635c297b0ef2dadab8ad08595ab6b393502057edb5a46771ca6751a78d234b not found: ID does not exist"
Apr 24 22:50:16.853792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.853759    2543 scope.go:117] "RemoveContainer" containerID="b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db"
Apr 24 22:50:16.853964 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:50:16.853949    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db\": container with ID starting with b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db not found: ID does not exist" containerID="b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db"
Apr 24 22:50:16.854002 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.853969    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db"} err="failed to get container status \"b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db\": rpc error: code = NotFound desc = could not find container \"b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db\": container with ID starting with b86afc6f2e21ebeab2fd23dc1517bad210ca2e4edb88ca1d20a7db7354df43db not found: ID does not exist"
Apr 24 22:50:16.858552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:16.858528    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-2kj85"]
Apr 24 22:50:17.055285 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:17.055259    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" path="/var/lib/kubelet/pods/a31e4385-23f1-4cd1-8cb1-1649a298d44c/volumes"
Apr 24 22:50:19.841952 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:19.841923    2543 generic.go:358] "Generic (PLEG): container finished" podID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerID="c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902" exitCode=0
Apr 24 22:50:19.842330 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:19.841976    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerDied","Data":"c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902"}
Apr 24 22:50:20.846198 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:20.846168    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerStarted","Data":"bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321"}
Apr 24 22:50:22.853866 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:22.853835    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerStarted","Data":"1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab"}
Apr 24 22:50:22.853866 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:22.853869    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerStarted","Data":"96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d"}
Apr 24 22:50:22.854263 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:22.854065    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:22.873556 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:22.873450    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podStartSLOduration=5.159277207 podStartE2EDuration="7.873436929s" podCreationTimestamp="2026-04-24 22:50:15 +0000 UTC" firstStartedPulling="2026-04-24 22:50:19.896775285 +0000 UTC m=+1223.323172793" lastFinishedPulling="2026-04-24 22:50:22.610934804 +0000 UTC m=+1226.037332515" observedRunningTime="2026-04-24 22:50:22.871598832 +0000 UTC m=+1226.297996374" watchObservedRunningTime="2026-04-24 22:50:22.873436929 +0000 UTC m=+1226.299834459"
Apr 24 22:50:23.859619 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:23.859575    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:23.859988 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:23.859629    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:29.867446 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:29.867420    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:50:49.870295 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:50:49.870264    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:51:29.871283 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:29.871253    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:51:35.251483 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.251434    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"]
Apr 24 22:51:35.251960 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.251906    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-container" containerID="cri-o://bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321" gracePeriod=30
Apr 24 22:51:35.252088 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.252060    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy" containerID="cri-o://1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab" gracePeriod=30
Apr 24 22:51:35.252169 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.252093    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-agent" containerID="cri-o://96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d" gracePeriod=30
Apr 24 22:51:35.315871 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.315845    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"]
Apr 24 22:51:35.316171 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.316159    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="kserve-container"
Apr 24 22:51:35.316212 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.316172    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="kserve-container"
Apr 24 22:51:35.316212 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.316183    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="storage-initializer"
Apr 24 22:51:35.316212 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.316189    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="storage-initializer"
Apr 24 22:51:35.316212 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.316196    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="kube-rbac-proxy"
Apr 24 22:51:35.316212 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.316201    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="kube-rbac-proxy"
Apr 24 22:51:35.316381 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.316253    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="kserve-container"
Apr 24 22:51:35.316381 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.316262    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="a31e4385-23f1-4cd1-8cb1-1649a298d44c" containerName="kube-rbac-proxy"
Apr 24 22:51:35.319450 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.319435    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.322556 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.322536    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Apr 24 22:51:35.322942 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.322924    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Apr 24 22:51:35.329680 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.329661    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"]
Apr 24 22:51:35.393459 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.393437    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlw6l\" (UniqueName: \"kubernetes.io/projected/c094066b-2e64-488d-939e-87290fa8e827-kube-api-access-dlw6l\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.393541 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.393490    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c094066b-2e64-488d-939e-87290fa8e827-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.393541 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.393513    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c094066b-2e64-488d-939e-87290fa8e827-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.393613 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.393553    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c094066b-2e64-488d-939e-87290fa8e827-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.494743 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.494713    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c094066b-2e64-488d-939e-87290fa8e827-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.494909 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.494754    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c094066b-2e64-488d-939e-87290fa8e827-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.494909 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.494784    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c094066b-2e64-488d-939e-87290fa8e827-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.494909 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.494850    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw6l\" (UniqueName: \"kubernetes.io/projected/c094066b-2e64-488d-939e-87290fa8e827-kube-api-access-dlw6l\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.495301 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.495279    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c094066b-2e64-488d-939e-87290fa8e827-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.495477 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.495450    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c094066b-2e64-488d-939e-87290fa8e827-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.497389 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.497365    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c094066b-2e64-488d-939e-87290fa8e827-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.503018 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.502956    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlw6l\" (UniqueName: \"kubernetes.io/projected/c094066b-2e64-488d-939e-87290fa8e827-kube-api-access-dlw6l\") pod \"isvc-paddle-predictor-6b8b7cfb4b-gx9xl\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.629966 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.629931    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:35.755255 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:35.755181    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"]
Apr 24 22:51:35.758383 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:51:35.758329    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc094066b_2e64_488d_939e_87290fa8e827.slice/crio-738b6f88e2914682442c8ffd47df5e070738e2e38fb3601e145c6e5a07d117f6 WatchSource:0}: Error finding container 738b6f88e2914682442c8ffd47df5e070738e2e38fb3601e145c6e5a07d117f6: Status 404 returned error can't find the container with id 738b6f88e2914682442c8ffd47df5e070738e2e38fb3601e145c6e5a07d117f6
Apr 24 22:51:36.071125 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:36.071023    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" event={"ID":"c094066b-2e64-488d-939e-87290fa8e827","Type":"ContainerStarted","Data":"ac300e4b466e3e0798375b0961fa0c1441a58c391f4b3c440744d681c284234e"}
Apr 24 22:51:36.071125 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:36.071072    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" event={"ID":"c094066b-2e64-488d-939e-87290fa8e827","Type":"ContainerStarted","Data":"738b6f88e2914682442c8ffd47df5e070738e2e38fb3601e145c6e5a07d117f6"}
Apr 24 22:51:36.073465 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:36.073435    2543 generic.go:358] "Generic (PLEG): container finished" podID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerID="1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab" exitCode=2
Apr 24 22:51:36.073582 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:36.073487    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerDied","Data":"1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab"}
Apr 24 22:51:37.078118 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:37.078093    2543 generic.go:358] "Generic (PLEG): container finished" podID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerID="bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321" exitCode=0
Apr 24 22:51:37.078476 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:37.078164    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerDied","Data":"bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321"}
Apr 24 22:51:39.863285 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:39.863243    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Apr 24 22:51:39.868293 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:39.868261    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.30:8080: connect: connection refused"
Apr 24 22:51:40.087069 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:40.087046    2543 generic.go:358] "Generic (PLEG): container finished" podID="c094066b-2e64-488d-939e-87290fa8e827" containerID="ac300e4b466e3e0798375b0961fa0c1441a58c391f4b3c440744d681c284234e" exitCode=0
Apr 24 22:51:40.087165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:40.087113    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" event={"ID":"c094066b-2e64-488d-939e-87290fa8e827","Type":"ContainerDied","Data":"ac300e4b466e3e0798375b0961fa0c1441a58c391f4b3c440744d681c284234e"}
Apr 24 22:51:44.863389 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:44.863352    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Apr 24 22:51:49.862802 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:49.862754    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Apr 24 22:51:49.863198 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:49.862926    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:51:49.868592 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:49.868557    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.30:8080: connect: connection refused"
Apr 24 22:51:51.123502 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:51.123463    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" event={"ID":"c094066b-2e64-488d-939e-87290fa8e827","Type":"ContainerStarted","Data":"e6ad5cfbda1cc779668e8794b2d7748c68bd028ba3629d755a6e46a820b2d0f6"}
Apr 24 22:51:51.123502 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:51.123509    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" event={"ID":"c094066b-2e64-488d-939e-87290fa8e827","Type":"ContainerStarted","Data":"e6aed1604f37b3265ab0bf29803943169194bdf299075c1b5165d1939a8e0ee9"}
Apr 24 22:51:51.123882 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:51.123790    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:51.124060 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:51.123921    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:51.125258 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:51.125232    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Apr 24 22:51:51.143581 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:51.143510    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podStartSLOduration=5.329350387 podStartE2EDuration="16.143498054s" podCreationTimestamp="2026-04-24 22:51:35 +0000 UTC" firstStartedPulling="2026-04-24 22:51:40.088250281 +0000 UTC m=+1303.514647789" lastFinishedPulling="2026-04-24 22:51:50.902397944 +0000 UTC m=+1314.328795456" observedRunningTime="2026-04-24 22:51:51.14159599 +0000 UTC m=+1314.567993519" watchObservedRunningTime="2026-04-24 22:51:51.143498054 +0000 UTC m=+1314.569895584"
Apr 24 22:51:52.127171 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:52.127133    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Apr 24 22:51:54.862562 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:54.862524    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Apr 24 22:51:57.131595 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:57.131563    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:51:57.132187 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:57.132163    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Apr 24 22:51:59.863327 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:59.863292    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Apr 24 22:51:59.868827 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:59.868805    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.30:8080: connect: connection refused"
Apr 24 22:51:59.868929 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:51:59.868914    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:52:04.863071 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:04.863030    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Apr 24 22:52:05.461191 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.461169    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:52:05.536831 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.536761    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-proxy-tls\") pod \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") "
Apr 24 22:52:05.536831 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.536820    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kserve-provision-location\") pod \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") "
Apr 24 22:52:05.537036 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.536853    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnbks\" (UniqueName: \"kubernetes.io/projected/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kube-api-access-rnbks\") pod \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") "
Apr 24 22:52:05.537036 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.536890    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\" (UID: \"0c3fd6b3-d5ba-4417-97ff-777aadcd9556\") "
Apr 24 22:52:05.537148 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.537087    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c3fd6b3-d5ba-4417-97ff-777aadcd9556" (UID: "0c3fd6b3-d5ba-4417-97ff-777aadcd9556"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:52:05.537268 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.537239    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "0c3fd6b3-d5ba-4417-97ff-777aadcd9556" (UID: "0c3fd6b3-d5ba-4417-97ff-777aadcd9556"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:52:05.538951 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.538927    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kube-api-access-rnbks" (OuterVolumeSpecName: "kube-api-access-rnbks") pod "0c3fd6b3-d5ba-4417-97ff-777aadcd9556" (UID: "0c3fd6b3-d5ba-4417-97ff-777aadcd9556"). InnerVolumeSpecName "kube-api-access-rnbks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:52:05.539104 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.538949    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c3fd6b3-d5ba-4417-97ff-777aadcd9556" (UID: "0c3fd6b3-d5ba-4417-97ff-777aadcd9556"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:52:05.637307 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.637275    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:52:05.637307 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.637300    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:52:05.637307 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.637310    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:52:05.637525 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:05.637320    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnbks\" (UniqueName: \"kubernetes.io/projected/0c3fd6b3-d5ba-4417-97ff-777aadcd9556-kube-api-access-rnbks\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:52:06.166004 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.165972    2543 generic.go:358] "Generic (PLEG): container finished" podID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerID="96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d" exitCode=137
Apr 24 22:52:06.166400 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.166013    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerDied","Data":"96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d"}
Apr 24 22:52:06.166400 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.166045    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx" event={"ID":"0c3fd6b3-d5ba-4417-97ff-777aadcd9556","Type":"ContainerDied","Data":"797c3af81e6193cc05cd6bbf15666b48f4f94e1b1bc13ae188c31fe3c7bd4dcb"}
Apr 24 22:52:06.166400 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.166057    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"
Apr 24 22:52:06.166400 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.166061    2543 scope.go:117] "RemoveContainer" containerID="1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab"
Apr 24 22:52:06.174654 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.174635    2543 scope.go:117] "RemoveContainer" containerID="96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d"
Apr 24 22:52:06.181751 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.181730    2543 scope.go:117] "RemoveContainer" containerID="bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321"
Apr 24 22:52:06.186795 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.186773    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"]
Apr 24 22:52:06.189498 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.189476    2543 scope.go:117] "RemoveContainer" containerID="c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902"
Apr 24 22:52:06.190713 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.190694    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-c4b9ff587-62qtx"]
Apr 24 22:52:06.196101 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.196086    2543 scope.go:117] "RemoveContainer" containerID="1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab"
Apr 24 22:52:06.196332 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:52:06.196314    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab\": container with ID starting with 1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab not found: ID does not exist" containerID="1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab"
Apr 24 22:52:06.196332 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.196356    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab"} err="failed to get container status \"1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab\": rpc error: code = NotFound desc = could not find container \"1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab\": container with ID starting with 1a48a3dd6901727d5db668cd55dedac314fe5c93bc824e0efa1cb74f644a73ab not found: ID does not exist"
Apr 24 22:52:06.196332 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.196391    2543 scope.go:117] "RemoveContainer" containerID="96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d"
Apr 24 22:52:06.196631 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:52:06.196610    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d\": container with ID starting with 96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d not found: ID does not exist" containerID="96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d"
Apr 24 22:52:06.196670 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.196627    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d"} err="failed to get container status \"96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d\": rpc error: code = NotFound desc = could not find container \"96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d\": container with ID starting with 96aa8de3c9ea741beeb71f8ae54f65421ae44106da41705e089108a7a492287d not found: ID does not exist"
Apr 24 22:52:06.196670 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.196640    2543 scope.go:117] "RemoveContainer" containerID="bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321"
Apr 24 22:52:06.196878 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:52:06.196862    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321\": container with ID starting with bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321 not found: ID does not exist" containerID="bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321"
Apr 24 22:52:06.196913 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.196886    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321"} err="failed to get container status \"bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321\": rpc error: code = NotFound desc = could not find container \"bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321\": container with ID starting with bbcd83034e3116eea270d28de5be853a71a1ed9d904ab37c06c4d1c872be1321 not found: ID does not exist"
Apr 24 22:52:06.196950 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.196902    2543 scope.go:117] "RemoveContainer" containerID="c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902"
Apr 24 22:52:06.197136 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:52:06.197119    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902\": container with ID starting with c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902 not found: ID does not exist" containerID="c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902"
Apr 24 22:52:06.197179 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:06.197142    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902"} err="failed to get container status \"c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902\": rpc error: code = NotFound desc = could not find container \"c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902\": container with ID starting with c14a8f5150e8e420d6706852e760faff7cfea6c8f4c62108ce5e0907168ee902 not found: ID does not exist"
Apr 24 22:52:07.054954 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:07.054877    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" path="/var/lib/kubelet/pods/0c3fd6b3-d5ba-4417-97ff-777aadcd9556/volumes"
Apr 24 22:52:07.132426 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:07.132391    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Apr 24 22:52:17.132986 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:17.132941    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Apr 24 22:52:27.132480 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:27.132440    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Apr 24 22:52:37.133494 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:37.133456    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:52:46.763534 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.763501    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"]
Apr 24 22:52:46.763975 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.763813    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container" containerID="cri-o://e6aed1604f37b3265ab0bf29803943169194bdf299075c1b5165d1939a8e0ee9" gracePeriod=30
Apr 24 22:52:46.763975 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.763857    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kube-rbac-proxy" containerID="cri-o://e6ad5cfbda1cc779668e8794b2d7748c68bd028ba3629d755a6e46a820b2d0f6" gracePeriod=30
Apr 24 22:52:46.850238 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850207    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"]
Apr 24 22:52:46.850544 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850530    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy"
Apr 24 22:52:46.850592 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850548    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy"
Apr 24 22:52:46.850592 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850564    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-container"
Apr 24 22:52:46.850592 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850570    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-container"
Apr 24 22:52:46.850592 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850582    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-agent"
Apr 24 22:52:46.850592 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850588    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-agent"
Apr 24 22:52:46.850738 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850597    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="storage-initializer"
Apr 24 22:52:46.850738 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850603    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="storage-initializer"
Apr 24 22:52:46.850738 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850654    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-agent"
Apr 24 22:52:46.850738 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850663    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kserve-container"
Apr 24 22:52:46.850738 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.850671    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="0c3fd6b3-d5ba-4417-97ff-777aadcd9556" containerName="kube-rbac-proxy"
Apr 24 22:52:46.853544 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.853529    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:46.855877 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.855859    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Apr 24 22:52:46.855877 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.855868    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Apr 24 22:52:46.861882 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.861862    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"]
Apr 24 22:52:46.936692 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.936671    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fcd0e9-19be-4694-949d-fad952ce52ee-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:46.936790 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.936718    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fcd0e9-19be-4694-949d-fad952ce52ee-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:46.936790 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.936748    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fcd0e9-19be-4694-949d-fad952ce52ee-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:46.936790 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:46.936774    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkq25\" (UniqueName: \"kubernetes.io/projected/06fcd0e9-19be-4694-949d-fad952ce52ee-kube-api-access-lkq25\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.037258 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.037197    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lkq25\" (UniqueName: \"kubernetes.io/projected/06fcd0e9-19be-4694-949d-fad952ce52ee-kube-api-access-lkq25\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.037258 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.037253    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fcd0e9-19be-4694-949d-fad952ce52ee-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.037411 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.037292    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fcd0e9-19be-4694-949d-fad952ce52ee-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.037411 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.037316    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fcd0e9-19be-4694-949d-fad952ce52ee-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.037502 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:52:47.037450    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Apr 24 22:52:47.037544 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:52:47.037517    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06fcd0e9-19be-4694-949d-fad952ce52ee-proxy-tls podName:06fcd0e9-19be-4694-949d-fad952ce52ee nodeName:}" failed. No retries permitted until 2026-04-24 22:52:47.53750015 +0000 UTC m=+1370.963897657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/06fcd0e9-19be-4694-949d-fad952ce52ee-proxy-tls") pod "isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" (UID: "06fcd0e9-19be-4694-949d-fad952ce52ee") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Apr 24 22:52:47.037769 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.037753    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fcd0e9-19be-4694-949d-fad952ce52ee-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.037955 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.037939    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fcd0e9-19be-4694-949d-fad952ce52ee-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.050360 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.050314    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkq25\" (UniqueName: \"kubernetes.io/projected/06fcd0e9-19be-4694-949d-fad952ce52ee-kube-api-access-lkq25\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.127710 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.127656    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Apr 24 22:52:47.132899 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.132869    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Apr 24 22:52:47.281718 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.281676    2543 generic.go:358] "Generic (PLEG): container finished" podID="c094066b-2e64-488d-939e-87290fa8e827" containerID="e6ad5cfbda1cc779668e8794b2d7748c68bd028ba3629d755a6e46a820b2d0f6" exitCode=2
Apr 24 22:52:47.281885 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.281754    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" event={"ID":"c094066b-2e64-488d-939e-87290fa8e827","Type":"ContainerDied","Data":"e6ad5cfbda1cc779668e8794b2d7748c68bd028ba3629d755a6e46a820b2d0f6"}
Apr 24 22:52:47.541507 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.541479    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fcd0e9-19be-4694-949d-fad952ce52ee-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.544042 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.544009    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fcd0e9-19be-4694-949d-fad952ce52ee-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.763512 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.763478    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:47.881275 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:47.881251    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"]
Apr 24 22:52:47.883836 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:52:47.883809    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fcd0e9_19be_4694_949d_fad952ce52ee.slice/crio-bb6bd093d33484814f96859a66f2ed0bc8a2f986356ebc5f8d8745a02279bfc8 WatchSource:0}: Error finding container bb6bd093d33484814f96859a66f2ed0bc8a2f986356ebc5f8d8745a02279bfc8: Status 404 returned error can't find the container with id bb6bd093d33484814f96859a66f2ed0bc8a2f986356ebc5f8d8745a02279bfc8
Apr 24 22:52:48.285902 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:48.285860    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" event={"ID":"06fcd0e9-19be-4694-949d-fad952ce52ee","Type":"ContainerStarted","Data":"c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25"}
Apr 24 22:52:48.285902 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:48.285897    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" event={"ID":"06fcd0e9-19be-4694-949d-fad952ce52ee","Type":"ContainerStarted","Data":"bb6bd093d33484814f96859a66f2ed0bc8a2f986356ebc5f8d8745a02279bfc8"}
Apr 24 22:52:49.290913 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.290879    2543 generic.go:358] "Generic (PLEG): container finished" podID="c094066b-2e64-488d-939e-87290fa8e827" containerID="e6aed1604f37b3265ab0bf29803943169194bdf299075c1b5165d1939a8e0ee9" exitCode=0
Apr 24 22:52:49.291229 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.290937    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" event={"ID":"c094066b-2e64-488d-939e-87290fa8e827","Type":"ContainerDied","Data":"e6aed1604f37b3265ab0bf29803943169194bdf299075c1b5165d1939a8e0ee9"}
Apr 24 22:52:49.397858 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.397838    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:52:49.459478 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.459439    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlw6l\" (UniqueName: \"kubernetes.io/projected/c094066b-2e64-488d-939e-87290fa8e827-kube-api-access-dlw6l\") pod \"c094066b-2e64-488d-939e-87290fa8e827\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") "
Apr 24 22:52:49.459642 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.459529    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c094066b-2e64-488d-939e-87290fa8e827-proxy-tls\") pod \"c094066b-2e64-488d-939e-87290fa8e827\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") "
Apr 24 22:52:49.459642 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.459558    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c094066b-2e64-488d-939e-87290fa8e827-kserve-provision-location\") pod \"c094066b-2e64-488d-939e-87290fa8e827\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") "
Apr 24 22:52:49.459642 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.459598    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c094066b-2e64-488d-939e-87290fa8e827-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"c094066b-2e64-488d-939e-87290fa8e827\" (UID: \"c094066b-2e64-488d-939e-87290fa8e827\") "
Apr 24 22:52:49.459951 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.459917    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c094066b-2e64-488d-939e-87290fa8e827-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "c094066b-2e64-488d-939e-87290fa8e827" (UID: "c094066b-2e64-488d-939e-87290fa8e827"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:52:49.461684 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.461632    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c094066b-2e64-488d-939e-87290fa8e827-kube-api-access-dlw6l" (OuterVolumeSpecName: "kube-api-access-dlw6l") pod "c094066b-2e64-488d-939e-87290fa8e827" (UID: "c094066b-2e64-488d-939e-87290fa8e827"). InnerVolumeSpecName "kube-api-access-dlw6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:52:49.461684 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.461634    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c094066b-2e64-488d-939e-87290fa8e827-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c094066b-2e64-488d-939e-87290fa8e827" (UID: "c094066b-2e64-488d-939e-87290fa8e827"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:52:49.469041 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.469021    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c094066b-2e64-488d-939e-87290fa8e827-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c094066b-2e64-488d-939e-87290fa8e827" (UID: "c094066b-2e64-488d-939e-87290fa8e827"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:52:49.560813 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.560793    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlw6l\" (UniqueName: \"kubernetes.io/projected/c094066b-2e64-488d-939e-87290fa8e827-kube-api-access-dlw6l\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:52:49.560813 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.560812    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c094066b-2e64-488d-939e-87290fa8e827-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:52:49.560922 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.560821    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c094066b-2e64-488d-939e-87290fa8e827-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:52:49.560922 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:49.560831    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c094066b-2e64-488d-939e-87290fa8e827-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:52:50.300049 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:50.300015    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl" event={"ID":"c094066b-2e64-488d-939e-87290fa8e827","Type":"ContainerDied","Data":"738b6f88e2914682442c8ffd47df5e070738e2e38fb3601e145c6e5a07d117f6"}
Apr 24 22:52:50.300504 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:50.300060    2543 scope.go:117] "RemoveContainer" containerID="e6ad5cfbda1cc779668e8794b2d7748c68bd028ba3629d755a6e46a820b2d0f6"
Apr 24 22:52:50.300504 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:50.300093    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"
Apr 24 22:52:50.308684 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:50.308666    2543 scope.go:117] "RemoveContainer" containerID="e6aed1604f37b3265ab0bf29803943169194bdf299075c1b5165d1939a8e0ee9"
Apr 24 22:52:50.315533 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:50.315517    2543 scope.go:117] "RemoveContainer" containerID="ac300e4b466e3e0798375b0961fa0c1441a58c391f4b3c440744d681c284234e"
Apr 24 22:52:50.321286 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:50.321265    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"]
Apr 24 22:52:50.324623 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:50.324602    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-gx9xl"]
Apr 24 22:52:51.054707 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:51.054675    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c094066b-2e64-488d-939e-87290fa8e827" path="/var/lib/kubelet/pods/c094066b-2e64-488d-939e-87290fa8e827/volumes"
Apr 24 22:52:52.307822 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:52.307788    2543 generic.go:358] "Generic (PLEG): container finished" podID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerID="c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25" exitCode=0
Apr 24 22:52:52.308197 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:52.307846    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" event={"ID":"06fcd0e9-19be-4694-949d-fad952ce52ee","Type":"ContainerDied","Data":"c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25"}
Apr 24 22:52:53.312766 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:53.312727    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" event={"ID":"06fcd0e9-19be-4694-949d-fad952ce52ee","Type":"ContainerStarted","Data":"f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9"}
Apr 24 22:52:53.313116 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:53.312773    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" event={"ID":"06fcd0e9-19be-4694-949d-fad952ce52ee","Type":"ContainerStarted","Data":"eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9"}
Apr 24 22:52:53.313116 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:53.312974    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:53.331277 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:53.331237    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podStartSLOduration=7.331226581 podStartE2EDuration="7.331226581s" podCreationTimestamp="2026-04-24 22:52:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:52:53.329756782 +0000 UTC m=+1376.756154312" watchObservedRunningTime="2026-04-24 22:52:53.331226581 +0000 UTC m=+1376.757624113"
Apr 24 22:52:54.316650 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:54.316618    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:52:54.317912 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:54.317888    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Apr 24 22:52:55.319543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:52:55.319500    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Apr 24 22:53:00.323985 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:00.323949    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:53:00.324593 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:00.324564    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Apr 24 22:53:10.324568 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:10.324529    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Apr 24 22:53:20.324950 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:20.324911    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Apr 24 22:53:30.324552 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:30.324512    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Apr 24 22:53:40.325530 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:40.325458    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:53:48.251973 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.251937    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"]
Apr 24 22:53:48.252462 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.252356    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container" containerID="cri-o://eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9" gracePeriod=30
Apr 24 22:53:48.252462 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.252406    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kube-rbac-proxy" containerID="cri-o://f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9" gracePeriod=30
Apr 24 22:53:48.341214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.341177    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"]
Apr 24 22:53:48.341486 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.341475    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="storage-initializer"
Apr 24 22:53:48.341544 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.341488    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="storage-initializer"
Apr 24 22:53:48.341544 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.341506    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container"
Apr 24 22:53:48.341544 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.341511    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container"
Apr 24 22:53:48.341544 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.341523    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kube-rbac-proxy"
Apr 24 22:53:48.341544 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.341528    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kube-rbac-proxy"
Apr 24 22:53:48.341699 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.341570    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kube-rbac-proxy"
Apr 24 22:53:48.341699 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.341581    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="c094066b-2e64-488d-939e-87290fa8e827" containerName="kserve-container"
Apr 24 22:53:48.344495 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.344479    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.347042 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.347026    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Apr 24 22:53:48.347108 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.347040    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Apr 24 22:53:48.354579 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.354559    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"]
Apr 24 22:53:48.400972 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.400947    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e537e0d-6099-41de-b083-6633d83df720-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.401073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.400989    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvzq\" (UniqueName: \"kubernetes.io/projected/8e537e0d-6099-41de-b083-6633d83df720-kube-api-access-nfvzq\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.401073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.401012    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e537e0d-6099-41de-b083-6633d83df720-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.401202 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.401074    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e537e0d-6099-41de-b083-6633d83df720-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.470324 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.470292    2543 generic.go:358] "Generic (PLEG): container finished" podID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerID="f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9" exitCode=2
Apr 24 22:53:48.470485 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.470368    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" event={"ID":"06fcd0e9-19be-4694-949d-fad952ce52ee","Type":"ContainerDied","Data":"f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9"}
Apr 24 22:53:48.501841 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.501813    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e537e0d-6099-41de-b083-6633d83df720-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.501950 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.501859    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvzq\" (UniqueName: \"kubernetes.io/projected/8e537e0d-6099-41de-b083-6633d83df720-kube-api-access-nfvzq\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.501950 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.501880    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e537e0d-6099-41de-b083-6633d83df720-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.501950 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.501907    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e537e0d-6099-41de-b083-6633d83df720-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.502116 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:53:48.502052    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Apr 24 22:53:48.502175 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:53:48.502123    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e537e0d-6099-41de-b083-6633d83df720-proxy-tls podName:8e537e0d-6099-41de-b083-6633d83df720 nodeName:}" failed. No retries permitted until 2026-04-24 22:53:49.002107097 +0000 UTC m=+1432.428504604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8e537e0d-6099-41de-b083-6633d83df720-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" (UID: "8e537e0d-6099-41de-b083-6633d83df720") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Apr 24 22:53:48.502262 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.502245    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e537e0d-6099-41de-b083-6633d83df720-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.502588 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.502571    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e537e0d-6099-41de-b083-6633d83df720-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:48.522649 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:48.522618    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvzq\" (UniqueName: \"kubernetes.io/projected/8e537e0d-6099-41de-b083-6633d83df720-kube-api-access-nfvzq\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:49.006530 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:49.006496    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e537e0d-6099-41de-b083-6633d83df720-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:49.009013 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:49.008992    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e537e0d-6099-41de-b083-6633d83df720-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:49.254526 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:49.254489    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:49.376814 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:49.376787    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"]
Apr 24 22:53:49.379595 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:53:49.379565    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e537e0d_6099_41de_b083_6633d83df720.slice/crio-6c07281f40499895b499b8470971fcfee77bfff4d9d1ebe555dd3d7775f5ce56 WatchSource:0}: Error finding container 6c07281f40499895b499b8470971fcfee77bfff4d9d1ebe555dd3d7775f5ce56: Status 404 returned error can't find the container with id 6c07281f40499895b499b8470971fcfee77bfff4d9d1ebe555dd3d7775f5ce56
Apr 24 22:53:49.381886 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:49.381871    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 22:53:49.474691 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:49.474651    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" event={"ID":"8e537e0d-6099-41de-b083-6633d83df720","Type":"ContainerStarted","Data":"fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191"}
Apr 24 22:53:49.474691 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:49.474694    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" event={"ID":"8e537e0d-6099-41de-b083-6633d83df720","Type":"ContainerStarted","Data":"6c07281f40499895b499b8470971fcfee77bfff4d9d1ebe555dd3d7775f5ce56"}
Apr 24 22:53:50.320290 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:50.320239    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Apr 24 22:53:50.324593 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:50.324557    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Apr 24 22:53:50.986660 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:50.986640    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:53:51.021065 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.021038    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fcd0e9-19be-4694-949d-fad952ce52ee-proxy-tls\") pod \"06fcd0e9-19be-4694-949d-fad952ce52ee\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") "
Apr 24 22:53:51.021201 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.021159    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fcd0e9-19be-4694-949d-fad952ce52ee-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"06fcd0e9-19be-4694-949d-fad952ce52ee\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") "
Apr 24 22:53:51.021262 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.021198    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fcd0e9-19be-4694-949d-fad952ce52ee-kserve-provision-location\") pod \"06fcd0e9-19be-4694-949d-fad952ce52ee\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") "
Apr 24 22:53:51.021262 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.021227    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkq25\" (UniqueName: \"kubernetes.io/projected/06fcd0e9-19be-4694-949d-fad952ce52ee-kube-api-access-lkq25\") pod \"06fcd0e9-19be-4694-949d-fad952ce52ee\" (UID: \"06fcd0e9-19be-4694-949d-fad952ce52ee\") "
Apr 24 22:53:51.021556 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.021532    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fcd0e9-19be-4694-949d-fad952ce52ee-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "06fcd0e9-19be-4694-949d-fad952ce52ee" (UID: "06fcd0e9-19be-4694-949d-fad952ce52ee"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:53:51.023150 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.023129    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06fcd0e9-19be-4694-949d-fad952ce52ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06fcd0e9-19be-4694-949d-fad952ce52ee" (UID: "06fcd0e9-19be-4694-949d-fad952ce52ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:53:51.023300 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.023282    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fcd0e9-19be-4694-949d-fad952ce52ee-kube-api-access-lkq25" (OuterVolumeSpecName: "kube-api-access-lkq25") pod "06fcd0e9-19be-4694-949d-fad952ce52ee" (UID: "06fcd0e9-19be-4694-949d-fad952ce52ee"). InnerVolumeSpecName "kube-api-access-lkq25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:53:51.036520 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.036494    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fcd0e9-19be-4694-949d-fad952ce52ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06fcd0e9-19be-4694-949d-fad952ce52ee" (UID: "06fcd0e9-19be-4694-949d-fad952ce52ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:53:51.121954 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.121925    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06fcd0e9-19be-4694-949d-fad952ce52ee-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:53:51.121954 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.121949    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06fcd0e9-19be-4694-949d-fad952ce52ee-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:53:51.122089 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.121960    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkq25\" (UniqueName: \"kubernetes.io/projected/06fcd0e9-19be-4694-949d-fad952ce52ee-kube-api-access-lkq25\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:53:51.122089 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.121969    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06fcd0e9-19be-4694-949d-fad952ce52ee-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:53:51.482429 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.482396    2543 generic.go:358] "Generic (PLEG): container finished" podID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerID="eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9" exitCode=0
Apr 24 22:53:51.482814 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.482472    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" event={"ID":"06fcd0e9-19be-4694-949d-fad952ce52ee","Type":"ContainerDied","Data":"eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9"}
Apr 24 22:53:51.482814 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.482481    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"
Apr 24 22:53:51.482814 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.482509    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx" event={"ID":"06fcd0e9-19be-4694-949d-fad952ce52ee","Type":"ContainerDied","Data":"bb6bd093d33484814f96859a66f2ed0bc8a2f986356ebc5f8d8745a02279bfc8"}
Apr 24 22:53:51.482814 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.482528    2543 scope.go:117] "RemoveContainer" containerID="f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9"
Apr 24 22:53:51.490482 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.490453    2543 scope.go:117] "RemoveContainer" containerID="eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9"
Apr 24 22:53:51.497310 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.497292    2543 scope.go:117] "RemoveContainer" containerID="c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25"
Apr 24 22:53:51.499914 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.499884    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"]
Apr 24 22:53:51.505289 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.505272    2543 scope.go:117] "RemoveContainer" containerID="f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9"
Apr 24 22:53:51.505546 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:53:51.505528    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9\": container with ID starting with f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9 not found: ID does not exist" containerID="f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9"
Apr 24 22:53:51.505619 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.505553    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9"} err="failed to get container status \"f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9\": rpc error: code = NotFound desc = could not find container \"f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9\": container with ID starting with f05f3892d8120f3d7656ff574ed06375958b98b43c259e0dc02bb40ba06baee9 not found: ID does not exist"
Apr 24 22:53:51.505619 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.505571    2543 scope.go:117] "RemoveContainer" containerID="eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9"
Apr 24 22:53:51.505835 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:53:51.505820    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9\": container with ID starting with eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9 not found: ID does not exist" containerID="eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9"
Apr 24 22:53:51.505885 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.505838    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9"} err="failed to get container status \"eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9\": rpc error: code = NotFound desc = could not find container \"eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9\": container with ID starting with eb43a4a103ce05f75df9b3c09d0c4b0eed271321ef1ba15684b8e5d5d3a741c9 not found: ID does not exist"
Apr 24 22:53:51.505885 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.505849    2543 scope.go:117] "RemoveContainer" containerID="c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25"
Apr 24 22:53:51.506081 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:53:51.506055    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25\": container with ID starting with c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25 not found: ID does not exist" containerID="c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25"
Apr 24 22:53:51.506126 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.506086    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25"} err="failed to get container status \"c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25\": rpc error: code = NotFound desc = could not find container \"c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25\": container with ID starting with c329856143e2ecf989d8327e2399e6c69f2b44aece8176b9a5498f11367b2d25 not found: ID does not exist"
Apr 24 22:53:51.506196 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:51.506180    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-bkqbx"]
Apr 24 22:53:53.054719 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:53.054684    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" path="/var/lib/kubelet/pods/06fcd0e9-19be-4694-949d-fad952ce52ee/volumes"
Apr 24 22:53:54.493880 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:54.493798    2543 generic.go:358] "Generic (PLEG): container finished" podID="8e537e0d-6099-41de-b083-6633d83df720" containerID="fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191" exitCode=0
Apr 24 22:53:54.493880 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:54.493866    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" event={"ID":"8e537e0d-6099-41de-b083-6633d83df720","Type":"ContainerDied","Data":"fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191"}
Apr 24 22:53:55.498638 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:55.498591    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" event={"ID":"8e537e0d-6099-41de-b083-6633d83df720","Type":"ContainerStarted","Data":"3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb"}
Apr 24 22:53:55.499038 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:55.498654    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" event={"ID":"8e537e0d-6099-41de-b083-6633d83df720","Type":"ContainerStarted","Data":"70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627"}
Apr 24 22:53:55.499038 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:55.498869    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:55.516937 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:55.516886    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" podStartSLOduration=7.51686944 podStartE2EDuration="7.51686944s" podCreationTimestamp="2026-04-24 22:53:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:53:55.515541506 +0000 UTC m=+1438.941939038" watchObservedRunningTime="2026-04-24 22:53:55.51686944 +0000 UTC m=+1438.943266971"
Apr 24 22:53:56.501467 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:56.501436    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:53:56.502719 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:56.502691    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 24 22:53:57.504265 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:53:57.504233    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 24 22:54:02.508354 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:02.508308    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:54:02.508885 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:02.508858    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 24 22:54:12.509398 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:12.509355    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 24 22:54:22.508847 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:22.508811    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 24 22:54:32.509593 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:32.509547    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Apr 24 22:54:42.509772 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:42.509746    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:54:49.958722 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:49.958691    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"]
Apr 24 22:54:49.959102 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:49.958978    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container" containerID="cri-o://70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627" gracePeriod=30
Apr 24 22:54:49.959102 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:49.959032    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kube-rbac-proxy" containerID="cri-o://3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb" gracePeriod=30
Apr 24 22:54:50.054894 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.054862    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"]
Apr 24 22:54:50.055179 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.055167    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container"
Apr 24 22:54:50.055237 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.055180    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container"
Apr 24 22:54:50.055237 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.055202    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="storage-initializer"
Apr 24 22:54:50.055237 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.055207    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="storage-initializer"
Apr 24 22:54:50.055237 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.055214    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kube-rbac-proxy"
Apr 24 22:54:50.055237 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.055220    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kube-rbac-proxy"
Apr 24 22:54:50.055413 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.055265    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kube-rbac-proxy"
Apr 24 22:54:50.055413 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.055274    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="06fcd0e9-19be-4694-949d-fad952ce52ee" containerName="kserve-container"
Apr 24 22:54:50.058395 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.058379    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.060684 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.060662    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Apr 24 22:54:50.060790 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.060704    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Apr 24 22:54:50.067888 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.067868    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"]
Apr 24 22:54:50.079254 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.079227    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/091aa62c-8f97-4994-8cce-0c81fcd1867d-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.079358 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.079278    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/091aa62c-8f97-4994-8cce-0c81fcd1867d-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.079410 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.079328    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091aa62c-8f97-4994-8cce-0c81fcd1867d-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.079459 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.079413    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4s6s\" (UniqueName: \"kubernetes.io/projected/091aa62c-8f97-4994-8cce-0c81fcd1867d-kube-api-access-v4s6s\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.179845 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.179809    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4s6s\" (UniqueName: \"kubernetes.io/projected/091aa62c-8f97-4994-8cce-0c81fcd1867d-kube-api-access-v4s6s\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.179978 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.179868    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/091aa62c-8f97-4994-8cce-0c81fcd1867d-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.179978 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.179905    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/091aa62c-8f97-4994-8cce-0c81fcd1867d-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.179978 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.179937    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091aa62c-8f97-4994-8cce-0c81fcd1867d-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.180283 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.180261    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091aa62c-8f97-4994-8cce-0c81fcd1867d-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.180548 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.180525    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/091aa62c-8f97-4994-8cce-0c81fcd1867d-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.182320 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.182293    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/091aa62c-8f97-4994-8cce-0c81fcd1867d-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.187859 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.187841    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4s6s\" (UniqueName: \"kubernetes.io/projected/091aa62c-8f97-4994-8cce-0c81fcd1867d-kube-api-access-v4s6s\") pod \"isvc-pmml-predictor-8bb578669-sn85p\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.368883 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.368812    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:54:50.485612 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.485582    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"]
Apr 24 22:54:50.488742 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:54:50.488716    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091aa62c_8f97_4994_8cce_0c81fcd1867d.slice/crio-45f61f94d2a70f375778189f819611db32713f2337080dcf8c53b0b6f484e427 WatchSource:0}: Error finding container 45f61f94d2a70f375778189f819611db32713f2337080dcf8c53b0b6f484e427: Status 404 returned error can't find the container with id 45f61f94d2a70f375778189f819611db32713f2337080dcf8c53b0b6f484e427
Apr 24 22:54:50.654398 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.654295    2543 generic.go:358] "Generic (PLEG): container finished" podID="8e537e0d-6099-41de-b083-6633d83df720" containerID="3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb" exitCode=2
Apr 24 22:54:50.654398 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.654370    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" event={"ID":"8e537e0d-6099-41de-b083-6633d83df720","Type":"ContainerDied","Data":"3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb"}
Apr 24 22:54:50.655778 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.655744    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" event={"ID":"091aa62c-8f97-4994-8cce-0c81fcd1867d","Type":"ContainerStarted","Data":"f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7"}
Apr 24 22:54:50.655914 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:50.655786    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" event={"ID":"091aa62c-8f97-4994-8cce-0c81fcd1867d","Type":"ContainerStarted","Data":"45f61f94d2a70f375778189f819611db32713f2337080dcf8c53b0b6f484e427"}
Apr 24 22:54:52.397275 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.397240    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:54:52.493014 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.492950    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e537e0d-6099-41de-b083-6633d83df720-proxy-tls\") pod \"8e537e0d-6099-41de-b083-6633d83df720\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") "
Apr 24 22:54:52.493014 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.492999    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e537e0d-6099-41de-b083-6633d83df720-kserve-provision-location\") pod \"8e537e0d-6099-41de-b083-6633d83df720\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") "
Apr 24 22:54:52.493167 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.493031    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfvzq\" (UniqueName: \"kubernetes.io/projected/8e537e0d-6099-41de-b083-6633d83df720-kube-api-access-nfvzq\") pod \"8e537e0d-6099-41de-b083-6633d83df720\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") "
Apr 24 22:54:52.493167 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.493061    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e537e0d-6099-41de-b083-6633d83df720-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"8e537e0d-6099-41de-b083-6633d83df720\" (UID: \"8e537e0d-6099-41de-b083-6633d83df720\") "
Apr 24 22:54:52.493473 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.493441    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e537e0d-6099-41de-b083-6633d83df720-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "8e537e0d-6099-41de-b083-6633d83df720" (UID: "8e537e0d-6099-41de-b083-6633d83df720"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:54:52.495040 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.495022    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e537e0d-6099-41de-b083-6633d83df720-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e537e0d-6099-41de-b083-6633d83df720" (UID: "8e537e0d-6099-41de-b083-6633d83df720"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:54:52.495115 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.495099    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e537e0d-6099-41de-b083-6633d83df720-kube-api-access-nfvzq" (OuterVolumeSpecName: "kube-api-access-nfvzq") pod "8e537e0d-6099-41de-b083-6633d83df720" (UID: "8e537e0d-6099-41de-b083-6633d83df720"). InnerVolumeSpecName "kube-api-access-nfvzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:54:52.508148 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.508124    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e537e0d-6099-41de-b083-6633d83df720-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e537e0d-6099-41de-b083-6633d83df720" (UID: "8e537e0d-6099-41de-b083-6633d83df720"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:54:52.594109 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.594086    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e537e0d-6099-41de-b083-6633d83df720-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:54:52.594109 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.594107    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e537e0d-6099-41de-b083-6633d83df720-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:54:52.594226 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.594117    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvzq\" (UniqueName: \"kubernetes.io/projected/8e537e0d-6099-41de-b083-6633d83df720-kube-api-access-nfvzq\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:54:52.594226 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.594126    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e537e0d-6099-41de-b083-6633d83df720-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:54:52.661985 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.661962    2543 generic.go:358] "Generic (PLEG): container finished" podID="8e537e0d-6099-41de-b083-6633d83df720" containerID="70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627" exitCode=0
Apr 24 22:54:52.662068 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.662032    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"
Apr 24 22:54:52.662122 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.662031    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" event={"ID":"8e537e0d-6099-41de-b083-6633d83df720","Type":"ContainerDied","Data":"70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627"}
Apr 24 22:54:52.662163 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.662138    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx" event={"ID":"8e537e0d-6099-41de-b083-6633d83df720","Type":"ContainerDied","Data":"6c07281f40499895b499b8470971fcfee77bfff4d9d1ebe555dd3d7775f5ce56"}
Apr 24 22:54:52.662163 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.662153    2543 scope.go:117] "RemoveContainer" containerID="3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb"
Apr 24 22:54:52.669938 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.669920    2543 scope.go:117] "RemoveContainer" containerID="70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627"
Apr 24 22:54:52.676913 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.676897    2543 scope.go:117] "RemoveContainer" containerID="fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191"
Apr 24 22:54:52.682935 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.682914    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"]
Apr 24 22:54:52.684733 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.684713    2543 scope.go:117] "RemoveContainer" containerID="3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb"
Apr 24 22:54:52.684992 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:54:52.684966    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb\": container with ID starting with 3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb not found: ID does not exist" containerID="3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb"
Apr 24 22:54:52.685073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.684998    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb"} err="failed to get container status \"3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb\": rpc error: code = NotFound desc = could not find container \"3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb\": container with ID starting with 3f412a36cb79df47dbadad8151abb118721a6b5ed35355a6abe704169d09d2bb not found: ID does not exist"
Apr 24 22:54:52.685073 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.685018    2543 scope.go:117] "RemoveContainer" containerID="70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627"
Apr 24 22:54:52.685256 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:54:52.685238    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627\": container with ID starting with 70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627 not found: ID does not exist" containerID="70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627"
Apr 24 22:54:52.685303 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.685272    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627"} err="failed to get container status \"70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627\": rpc error: code = NotFound desc = could not find container \"70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627\": container with ID starting with 70e5d33bcd95a07e1752b6a31180d3d14bb8a507f4d616c0147b6bbb0b109627 not found: ID does not exist"
Apr 24 22:54:52.685303 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.685286    2543 scope.go:117] "RemoveContainer" containerID="fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191"
Apr 24 22:54:52.685570 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:54:52.685552    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191\": container with ID starting with fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191 not found: ID does not exist" containerID="fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191"
Apr 24 22:54:52.685632 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.685587    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191"} err="failed to get container status \"fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191\": rpc error: code = NotFound desc = could not find container \"fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191\": container with ID starting with fe89385cacf6c1a8047ad92a91a941526768dee148346094fd7195bc317a4191 not found: ID does not exist"
Apr 24 22:54:52.685701 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:52.685684    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-cfqdx"]
Apr 24 22:54:53.054945 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:53.054908    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e537e0d-6099-41de-b083-6633d83df720" path="/var/lib/kubelet/pods/8e537e0d-6099-41de-b083-6633d83df720/volumes"
Apr 24 22:54:54.669519 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:54.669490    2543 generic.go:358] "Generic (PLEG): container finished" podID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerID="f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7" exitCode=0
Apr 24 22:54:54.669894 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:54.669567    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" event={"ID":"091aa62c-8f97-4994-8cce-0c81fcd1867d","Type":"ContainerDied","Data":"f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7"}
Apr 24 22:54:57.466595 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:57.466564    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:54:57.467045 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:54:57.466669    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:55:02.697945 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:02.697908    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" event={"ID":"091aa62c-8f97-4994-8cce-0c81fcd1867d","Type":"ContainerStarted","Data":"c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d"}
Apr 24 22:55:02.697945 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:02.697946    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" event={"ID":"091aa62c-8f97-4994-8cce-0c81fcd1867d","Type":"ContainerStarted","Data":"ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304"}
Apr 24 22:55:02.698458 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:02.698143    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:55:02.715656 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:02.715603    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podStartSLOduration=5.668808025 podStartE2EDuration="12.715589111s" podCreationTimestamp="2026-04-24 22:54:50 +0000 UTC" firstStartedPulling="2026-04-24 22:54:54.670723575 +0000 UTC m=+1498.097121083" lastFinishedPulling="2026-04-24 22:55:01.717504661 +0000 UTC m=+1505.143902169" observedRunningTime="2026-04-24 22:55:02.715105523 +0000 UTC m=+1506.141503078" watchObservedRunningTime="2026-04-24 22:55:02.715589111 +0000 UTC m=+1506.141986642"
Apr 24 22:55:03.701121 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:03.701084    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:55:03.702015 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:03.701988    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:55:04.704231 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:04.704185    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:55:09.708130 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:09.708061    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:55:09.708649 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:09.708625    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:55:19.708826 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:19.708781    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:55:29.708952 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:29.708902    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:55:39.709019 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:39.708968    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:55:49.708650 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:49.708604    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:55:59.709528 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:55:59.709491    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:56:09.708645 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:09.708605    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:56:11.052385 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:11.052319    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Apr 24 22:56:21.055331 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:21.055302    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:56:31.422246 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.422166    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"]
Apr 24 22:56:31.422632 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.422489    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container" containerID="cri-o://ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304" gracePeriod=30
Apr 24 22:56:31.422632 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.422552    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kube-rbac-proxy" containerID="cri-o://c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d" gracePeriod=30
Apr 24 22:56:31.555131 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.555102    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"]
Apr 24 22:56:31.555453 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.555439    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="storage-initializer"
Apr 24 22:56:31.555502 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.555454    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="storage-initializer"
Apr 24 22:56:31.555502 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.555476    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kube-rbac-proxy"
Apr 24 22:56:31.555502 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.555481    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kube-rbac-proxy"
Apr 24 22:56:31.555502 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.555495    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container"
Apr 24 22:56:31.555625 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.555503    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container"
Apr 24 22:56:31.555625 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.555549    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kserve-container"
Apr 24 22:56:31.555625 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.555558    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e537e0d-6099-41de-b083-6633d83df720" containerName="kube-rbac-proxy"
Apr 24 22:56:31.558571 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.558557    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.561453 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.561434    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Apr 24 22:56:31.561550 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.561477    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Apr 24 22:56:31.568581 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.568562    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"]
Apr 24 22:56:31.635995 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.635971    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5758bf92-3ee8-4d0d-88ac-603b773508c8-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.636103 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.636005    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvbp\" (UniqueName: \"kubernetes.io/projected/5758bf92-3ee8-4d0d-88ac-603b773508c8-kube-api-access-xrvbp\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.636103 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.636030    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5758bf92-3ee8-4d0d-88ac-603b773508c8-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.636198 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.636107    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5758bf92-3ee8-4d0d-88ac-603b773508c8-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.736941 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.736916    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5758bf92-3ee8-4d0d-88ac-603b773508c8-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.737038 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.736975    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5758bf92-3ee8-4d0d-88ac-603b773508c8-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.737038 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.737004    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvbp\" (UniqueName: \"kubernetes.io/projected/5758bf92-3ee8-4d0d-88ac-603b773508c8-kube-api-access-xrvbp\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.737038 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.737027    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5758bf92-3ee8-4d0d-88ac-603b773508c8-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.737140 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:56:31.737037    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Apr 24 22:56:31.737140 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:56:31.737106    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5758bf92-3ee8-4d0d-88ac-603b773508c8-proxy-tls podName:5758bf92-3ee8-4d0d-88ac-603b773508c8 nodeName:}" failed. No retries permitted until 2026-04-24 22:56:32.237086011 +0000 UTC m=+1595.663483521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5758bf92-3ee8-4d0d-88ac-603b773508c8-proxy-tls") pod "isvc-pmml-runtime-predictor-67bc544947-f7qtd" (UID: "5758bf92-3ee8-4d0d-88ac-603b773508c8") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Apr 24 22:56:31.737386 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.737370    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5758bf92-3ee8-4d0d-88ac-603b773508c8-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.737636 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.737617    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5758bf92-3ee8-4d0d-88ac-603b773508c8-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.745596 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.745575    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvbp\" (UniqueName: \"kubernetes.io/projected/5758bf92-3ee8-4d0d-88ac-603b773508c8-kube-api-access-xrvbp\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:31.968754 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.968719    2543 generic.go:358] "Generic (PLEG): container finished" podID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerID="c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d" exitCode=2
Apr 24 22:56:31.968904 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:31.968792    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" event={"ID":"091aa62c-8f97-4994-8cce-0c81fcd1867d","Type":"ContainerDied","Data":"c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d"}
Apr 24 22:56:32.241233 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:32.241191    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5758bf92-3ee8-4d0d-88ac-603b773508c8-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:32.243677 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:32.243658    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5758bf92-3ee8-4d0d-88ac-603b773508c8-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-f7qtd\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:32.468813 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:32.468783    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:32.586911 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:32.586885    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"]
Apr 24 22:56:32.589538 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:56:32.589506    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5758bf92_3ee8_4d0d_88ac_603b773508c8.slice/crio-70863382940dddb7516f037edeed6d170de95d1b729562b8e82f173a0978d7e2 WatchSource:0}: Error finding container 70863382940dddb7516f037edeed6d170de95d1b729562b8e82f173a0978d7e2: Status 404 returned error can't find the container with id 70863382940dddb7516f037edeed6d170de95d1b729562b8e82f173a0978d7e2
Apr 24 22:56:32.973400 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:32.973369    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" event={"ID":"5758bf92-3ee8-4d0d-88ac-603b773508c8","Type":"ContainerStarted","Data":"370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52"}
Apr 24 22:56:32.973400 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:32.973407    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" event={"ID":"5758bf92-3ee8-4d0d-88ac-603b773508c8","Type":"ContainerStarted","Data":"70863382940dddb7516f037edeed6d170de95d1b729562b8e82f173a0978d7e2"}
Apr 24 22:56:34.562443 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.562420    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:56:34.662554 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.662523    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/091aa62c-8f97-4994-8cce-0c81fcd1867d-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"091aa62c-8f97-4994-8cce-0c81fcd1867d\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") "
Apr 24 22:56:34.662734 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.662587    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091aa62c-8f97-4994-8cce-0c81fcd1867d-kserve-provision-location\") pod \"091aa62c-8f97-4994-8cce-0c81fcd1867d\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") "
Apr 24 22:56:34.662734 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.662634    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4s6s\" (UniqueName: \"kubernetes.io/projected/091aa62c-8f97-4994-8cce-0c81fcd1867d-kube-api-access-v4s6s\") pod \"091aa62c-8f97-4994-8cce-0c81fcd1867d\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") "
Apr 24 22:56:34.662734 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.662675    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/091aa62c-8f97-4994-8cce-0c81fcd1867d-proxy-tls\") pod \"091aa62c-8f97-4994-8cce-0c81fcd1867d\" (UID: \"091aa62c-8f97-4994-8cce-0c81fcd1867d\") "
Apr 24 22:56:34.662914 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.662889    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091aa62c-8f97-4994-8cce-0c81fcd1867d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "091aa62c-8f97-4994-8cce-0c81fcd1867d" (UID: "091aa62c-8f97-4994-8cce-0c81fcd1867d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:56:34.662914 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.662901    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091aa62c-8f97-4994-8cce-0c81fcd1867d-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "091aa62c-8f97-4994-8cce-0c81fcd1867d" (UID: "091aa62c-8f97-4994-8cce-0c81fcd1867d"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:56:34.664785 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.664759    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/091aa62c-8f97-4994-8cce-0c81fcd1867d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "091aa62c-8f97-4994-8cce-0c81fcd1867d" (UID: "091aa62c-8f97-4994-8cce-0c81fcd1867d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:56:34.664878 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.664825    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091aa62c-8f97-4994-8cce-0c81fcd1867d-kube-api-access-v4s6s" (OuterVolumeSpecName: "kube-api-access-v4s6s") pod "091aa62c-8f97-4994-8cce-0c81fcd1867d" (UID: "091aa62c-8f97-4994-8cce-0c81fcd1867d"). InnerVolumeSpecName "kube-api-access-v4s6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:56:34.763636 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.763604    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/091aa62c-8f97-4994-8cce-0c81fcd1867d-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:56:34.763636 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.763637    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/091aa62c-8f97-4994-8cce-0c81fcd1867d-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:56:34.763780 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.763649    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/091aa62c-8f97-4994-8cce-0c81fcd1867d-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:56:34.763780 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.763659    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4s6s\" (UniqueName: \"kubernetes.io/projected/091aa62c-8f97-4994-8cce-0c81fcd1867d-kube-api-access-v4s6s\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:56:34.980332 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.980298    2543 generic.go:358] "Generic (PLEG): container finished" podID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerID="ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304" exitCode=0
Apr 24 22:56:34.980499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.980365    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" event={"ID":"091aa62c-8f97-4994-8cce-0c81fcd1867d","Type":"ContainerDied","Data":"ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304"}
Apr 24 22:56:34.980499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.980402    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p" event={"ID":"091aa62c-8f97-4994-8cce-0c81fcd1867d","Type":"ContainerDied","Data":"45f61f94d2a70f375778189f819611db32713f2337080dcf8c53b0b6f484e427"}
Apr 24 22:56:34.980499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.980406    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"
Apr 24 22:56:34.980499 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.980418    2543 scope.go:117] "RemoveContainer" containerID="c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d"
Apr 24 22:56:34.988914 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.988892    2543 scope.go:117] "RemoveContainer" containerID="ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304"
Apr 24 22:56:34.996063 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:34.996046    2543 scope.go:117] "RemoveContainer" containerID="f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7"
Apr 24 22:56:35.002220 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:35.002200    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"]
Apr 24 22:56:35.004243 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:35.004212    2543 scope.go:117] "RemoveContainer" containerID="c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d"
Apr 24 22:56:35.004587 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:56:35.004565    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d\": container with ID starting with c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d not found: ID does not exist" containerID="c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d"
Apr 24 22:56:35.004679 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:35.004597    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d"} err="failed to get container status \"c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d\": rpc error: code = NotFound desc = could not find container \"c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d\": container with ID starting with c819db53cda19580ec3bd2aabdd3675766917f10a8ea181b29674bc697e7e73d not found: ID does not exist"
Apr 24 22:56:35.004679 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:35.004619    2543 scope.go:117] "RemoveContainer" containerID="ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304"
Apr 24 22:56:35.004847 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:56:35.004830    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304\": container with ID starting with ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304 not found: ID does not exist" containerID="ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304"
Apr 24 22:56:35.004915 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:35.004854    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304"} err="failed to get container status \"ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304\": rpc error: code = NotFound desc = could not find container \"ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304\": container with ID starting with ea1d20033e9de0cda9ee4051b83774822ce0941b13862c92797308c02e1d1304 not found: ID does not exist"
Apr 24 22:56:35.004915 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:35.004876    2543 scope.go:117] "RemoveContainer" containerID="f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7"
Apr 24 22:56:35.005128 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:56:35.005111    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7\": container with ID starting with f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7 not found: ID does not exist" containerID="f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7"
Apr 24 22:56:35.005173 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:35.005133    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7"} err="failed to get container status \"f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7\": rpc error: code = NotFound desc = could not find container \"f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7\": container with ID starting with f26ab5a0043323b7ee6f0075d2bbac1b6a8bbff811da87858d1e45b6d3785ed7 not found: ID does not exist"
Apr 24 22:56:35.005770 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:35.005753    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-sn85p"]
Apr 24 22:56:35.054570 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:35.054548    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" path="/var/lib/kubelet/pods/091aa62c-8f97-4994-8cce-0c81fcd1867d/volumes"
Apr 24 22:56:36.988757 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:36.988726    2543 generic.go:358] "Generic (PLEG): container finished" podID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerID="370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52" exitCode=0
Apr 24 22:56:36.989147 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:36.988773    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" event={"ID":"5758bf92-3ee8-4d0d-88ac-603b773508c8","Type":"ContainerDied","Data":"370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52"}
Apr 24 22:56:37.993365 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:37.993318    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" event={"ID":"5758bf92-3ee8-4d0d-88ac-603b773508c8","Type":"ContainerStarted","Data":"64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e"}
Apr 24 22:56:37.993741 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:37.993373    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" event={"ID":"5758bf92-3ee8-4d0d-88ac-603b773508c8","Type":"ContainerStarted","Data":"c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8"}
Apr 24 22:56:37.993741 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:37.993702    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:37.993863 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:37.993844    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:37.995145 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:37.995119    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:56:38.010784 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:38.010743    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podStartSLOduration=7.010714584 podStartE2EDuration="7.010714584s" podCreationTimestamp="2026-04-24 22:56:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:56:38.009035969 +0000 UTC m=+1601.435433498" watchObservedRunningTime="2026-04-24 22:56:38.010714584 +0000 UTC m=+1601.437112114"
Apr 24 22:56:39.001135 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:39.001093    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:56:40.001316 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:40.001270    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:56:45.005511 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:45.005483    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:56:45.006103 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:45.006077    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:56:55.006700 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:56:55.006661    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:57:05.006792 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:57:05.006755    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:57:15.006555 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:57:15.006515    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:57:25.006730 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:57:25.006692    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:57:35.006906 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:57:35.006864    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:57:45.006409 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:57:45.006368    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:57:55.007210 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:57:55.007182    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:58:02.459574 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.459543    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"]
Apr 24 22:58:02.460069 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.459861    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" containerID="cri-o://c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8" gracePeriod=30
Apr 24 22:58:02.460069 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.459881    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kube-rbac-proxy" containerID="cri-o://64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e" gracePeriod=30
Apr 24 22:58:02.573086 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.573051    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"]
Apr 24 22:58:02.573453 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.573437    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kube-rbac-proxy"
Apr 24 22:58:02.573543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.573455    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kube-rbac-proxy"
Apr 24 22:58:02.573543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.573478    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="storage-initializer"
Apr 24 22:58:02.573543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.573487    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="storage-initializer"
Apr 24 22:58:02.573543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.573509    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container"
Apr 24 22:58:02.573543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.573518    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container"
Apr 24 22:58:02.573801 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.573588    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kube-rbac-proxy"
Apr 24 22:58:02.573801 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.573601    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="091aa62c-8f97-4994-8cce-0c81fcd1867d" containerName="kserve-container"
Apr 24 22:58:02.576861 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.576841    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.579026 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.579005    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Apr 24 22:58:02.579125 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.579068    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Apr 24 22:58:02.584602 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.584580    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"]
Apr 24 22:58:02.722100 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.722034    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e27b2b-a351-4e06-8d3e-2f881a955071-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.722100 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.722093    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e27b2b-a351-4e06-8d3e-2f881a955071-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.722241 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.722148    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqvn\" (UniqueName: \"kubernetes.io/projected/77e27b2b-a351-4e06-8d3e-2f881a955071-kube-api-access-xtqvn\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.722241 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.722190    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e27b2b-a351-4e06-8d3e-2f881a955071-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.823121 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.823096    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e27b2b-a351-4e06-8d3e-2f881a955071-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.823244 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.823154    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e27b2b-a351-4e06-8d3e-2f881a955071-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.823244 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.823184    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqvn\" (UniqueName: \"kubernetes.io/projected/77e27b2b-a351-4e06-8d3e-2f881a955071-kube-api-access-xtqvn\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.823244 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.823206    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e27b2b-a351-4e06-8d3e-2f881a955071-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.823600 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.823566    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e27b2b-a351-4e06-8d3e-2f881a955071-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.824043 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.823894    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e27b2b-a351-4e06-8d3e-2f881a955071-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.825646 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.825627    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e27b2b-a351-4e06-8d3e-2f881a955071-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.830621 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.830587    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqvn\" (UniqueName: \"kubernetes.io/projected/77e27b2b-a351-4e06-8d3e-2f881a955071-kube-api-access-xtqvn\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:02.887912 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:02.887885    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:03.004165 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:03.004139    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"]
Apr 24 22:58:03.006630 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:58:03.006607    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e27b2b_a351_4e06_8d3e_2f881a955071.slice/crio-af62c5d04296bc3baddee4fd6dc78806d1a8298ad9721966bff0353a4473c70f WatchSource:0}: Error finding container af62c5d04296bc3baddee4fd6dc78806d1a8298ad9721966bff0353a4473c70f: Status 404 returned error can't find the container with id af62c5d04296bc3baddee4fd6dc78806d1a8298ad9721966bff0353a4473c70f
Apr 24 22:58:03.226185 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:03.226147    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" event={"ID":"77e27b2b-a351-4e06-8d3e-2f881a955071","Type":"ContainerStarted","Data":"38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a"}
Apr 24 22:58:03.226372 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:03.226191    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" event={"ID":"77e27b2b-a351-4e06-8d3e-2f881a955071","Type":"ContainerStarted","Data":"af62c5d04296bc3baddee4fd6dc78806d1a8298ad9721966bff0353a4473c70f"}
Apr 24 22:58:03.231168 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:03.231143    2543 generic.go:358] "Generic (PLEG): container finished" podID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerID="64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e" exitCode=2
Apr 24 22:58:03.231293 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:03.231219    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" event={"ID":"5758bf92-3ee8-4d0d-88ac-603b773508c8","Type":"ContainerDied","Data":"64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e"}
Apr 24 22:58:05.002260 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.002221    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Apr 24 22:58:05.006530 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.006509    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Apr 24 22:58:05.600733 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.600714    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:58:05.743994 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.743969    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5758bf92-3ee8-4d0d-88ac-603b773508c8-proxy-tls\") pod \"5758bf92-3ee8-4d0d-88ac-603b773508c8\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") "
Apr 24 22:58:05.744128 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.744009    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5758bf92-3ee8-4d0d-88ac-603b773508c8-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"5758bf92-3ee8-4d0d-88ac-603b773508c8\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") "
Apr 24 22:58:05.744128 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.744045    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5758bf92-3ee8-4d0d-88ac-603b773508c8-kserve-provision-location\") pod \"5758bf92-3ee8-4d0d-88ac-603b773508c8\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") "
Apr 24 22:58:05.744240 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.744219    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrvbp\" (UniqueName: \"kubernetes.io/projected/5758bf92-3ee8-4d0d-88ac-603b773508c8-kube-api-access-xrvbp\") pod \"5758bf92-3ee8-4d0d-88ac-603b773508c8\" (UID: \"5758bf92-3ee8-4d0d-88ac-603b773508c8\") "
Apr 24 22:58:05.744358 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.744318    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5758bf92-3ee8-4d0d-88ac-603b773508c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5758bf92-3ee8-4d0d-88ac-603b773508c8" (UID: "5758bf92-3ee8-4d0d-88ac-603b773508c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:58:05.744431 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.744402    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5758bf92-3ee8-4d0d-88ac-603b773508c8-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "5758bf92-3ee8-4d0d-88ac-603b773508c8" (UID: "5758bf92-3ee8-4d0d-88ac-603b773508c8"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:58:05.744545 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.744493    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5758bf92-3ee8-4d0d-88ac-603b773508c8-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:58:05.744545 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.744508    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5758bf92-3ee8-4d0d-88ac-603b773508c8-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:58:05.746256 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.746230    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5758bf92-3ee8-4d0d-88ac-603b773508c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5758bf92-3ee8-4d0d-88ac-603b773508c8" (UID: "5758bf92-3ee8-4d0d-88ac-603b773508c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:58:05.746256 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.746239    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5758bf92-3ee8-4d0d-88ac-603b773508c8-kube-api-access-xrvbp" (OuterVolumeSpecName: "kube-api-access-xrvbp") pod "5758bf92-3ee8-4d0d-88ac-603b773508c8" (UID: "5758bf92-3ee8-4d0d-88ac-603b773508c8"). InnerVolumeSpecName "kube-api-access-xrvbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:58:05.845511 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.845480    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrvbp\" (UniqueName: \"kubernetes.io/projected/5758bf92-3ee8-4d0d-88ac-603b773508c8-kube-api-access-xrvbp\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:58:05.845511 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:05.845508    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5758bf92-3ee8-4d0d-88ac-603b773508c8-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:58:06.241358 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.241310    2543 generic.go:358] "Generic (PLEG): container finished" podID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerID="c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8" exitCode=0
Apr 24 22:58:06.241759 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.241408    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"
Apr 24 22:58:06.241759 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.241403    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" event={"ID":"5758bf92-3ee8-4d0d-88ac-603b773508c8","Type":"ContainerDied","Data":"c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8"}
Apr 24 22:58:06.241759 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.241524    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd" event={"ID":"5758bf92-3ee8-4d0d-88ac-603b773508c8","Type":"ContainerDied","Data":"70863382940dddb7516f037edeed6d170de95d1b729562b8e82f173a0978d7e2"}
Apr 24 22:58:06.241759 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.241547    2543 scope.go:117] "RemoveContainer" containerID="64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e"
Apr 24 22:58:06.249534 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.249519    2543 scope.go:117] "RemoveContainer" containerID="c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8"
Apr 24 22:58:06.256251 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.256223    2543 scope.go:117] "RemoveContainer" containerID="370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52"
Apr 24 22:58:06.261291 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.261270    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"]
Apr 24 22:58:06.263683 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.263654    2543 scope.go:117] "RemoveContainer" containerID="64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e"
Apr 24 22:58:06.263946 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:58:06.263925    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e\": container with ID starting with 64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e not found: ID does not exist" containerID="64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e"
Apr 24 22:58:06.264014 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.263953    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e"} err="failed to get container status \"64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e\": rpc error: code = NotFound desc = could not find container \"64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e\": container with ID starting with 64851365fc5d944f2f1be8bd0a083fea1ced84dff252afab8b1f1b0af3c3594e not found: ID does not exist"
Apr 24 22:58:06.264014 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.263970    2543 scope.go:117] "RemoveContainer" containerID="c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8"
Apr 24 22:58:06.264212 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:58:06.264196    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8\": container with ID starting with c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8 not found: ID does not exist" containerID="c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8"
Apr 24 22:58:06.264259 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.264215    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8"} err="failed to get container status \"c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8\": rpc error: code = NotFound desc = could not find container \"c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8\": container with ID starting with c9663ec43d34f2f712250cf940c3923bd4f37364c28f34d0f5fb26d7670276d8 not found: ID does not exist"
Apr 24 22:58:06.264259 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.264229    2543 scope.go:117] "RemoveContainer" containerID="370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52"
Apr 24 22:58:06.264546 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:58:06.264526    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52\": container with ID starting with 370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52 not found: ID does not exist" containerID="370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52"
Apr 24 22:58:06.264607 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.264551    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52"} err="failed to get container status \"370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52\": rpc error: code = NotFound desc = could not find container \"370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52\": container with ID starting with 370dde4920465d58309bc6ba3ce05eb4169eb54b1b2edb5d852ec79c3d095d52 not found: ID does not exist"
Apr 24 22:58:06.265271 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:06.265253    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-f7qtd"]
Apr 24 22:58:07.055635 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:07.055565    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" path="/var/lib/kubelet/pods/5758bf92-3ee8-4d0d-88ac-603b773508c8/volumes"
Apr 24 22:58:07.245320 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:07.245287    2543 generic.go:358] "Generic (PLEG): container finished" podID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerID="38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a" exitCode=0
Apr 24 22:58:07.245718 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:07.245366    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" event={"ID":"77e27b2b-a351-4e06-8d3e-2f881a955071","Type":"ContainerDied","Data":"38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a"}
Apr 24 22:58:08.250383 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:08.250333    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" event={"ID":"77e27b2b-a351-4e06-8d3e-2f881a955071","Type":"ContainerStarted","Data":"e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92"}
Apr 24 22:58:08.250383 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:08.250388    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" event={"ID":"77e27b2b-a351-4e06-8d3e-2f881a955071","Type":"ContainerStarted","Data":"7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc"}
Apr 24 22:58:08.250842 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:08.250592    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:08.268261 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:08.268224    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podStartSLOduration=6.268212523 podStartE2EDuration="6.268212523s" podCreationTimestamp="2026-04-24 22:58:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:58:08.266994865 +0000 UTC m=+1691.693392395" watchObservedRunningTime="2026-04-24 22:58:08.268212523 +0000 UTC m=+1691.694610053"
Apr 24 22:58:09.253754 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:09.253720    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:09.254918 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:09.254888    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:58:10.255986 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:10.255948    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:58:15.259871 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:15.259835    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:58:15.260518 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:15.260489    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:58:25.260864 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:25.260818    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:58:35.260861 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:35.260825    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:58:45.260434 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:45.260392    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:58:55.261005 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:58:55.260970    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:59:05.261226 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:05.261189    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:59:15.261084 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:15.261038    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:59:25.261267 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:25.261230    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:59:33.659075 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.659045    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"]
Apr 24 22:59:33.659485 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.659369    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" containerID="cri-o://7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc" gracePeriod=30
Apr 24 22:59:33.659485 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.659431    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kube-rbac-proxy" containerID="cri-o://e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92" gracePeriod=30
Apr 24 22:59:33.778882 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.778856    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"]
Apr 24 22:59:33.779161 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.779150    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="storage-initializer"
Apr 24 22:59:33.779208 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.779163    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="storage-initializer"
Apr 24 22:59:33.779208 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.779179    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kube-rbac-proxy"
Apr 24 22:59:33.779208 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.779185    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kube-rbac-proxy"
Apr 24 22:59:33.779208 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.779192    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container"
Apr 24 22:59:33.779208 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.779197    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container"
Apr 24 22:59:33.779444 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.779247    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kserve-container"
Apr 24 22:59:33.779444 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.779256    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="5758bf92-3ee8-4d0d-88ac-603b773508c8" containerName="kube-rbac-proxy"
Apr 24 22:59:33.782174 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.782158    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.784422 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.784400    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-0c0f7e-predictor-serving-cert\""
Apr 24 22:59:33.784505 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.784401    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\""
Apr 24 22:59:33.789471 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.789451    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"]
Apr 24 22:59:33.880843 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.880815    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a910569-1554-43de-96ea-d238df69afb0-proxy-tls\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.880991 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.880850    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a910569-1554-43de-96ea-d238df69afb0-kserve-provision-location\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.880991 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.880878    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcgg\" (UniqueName: \"kubernetes.io/projected/6a910569-1554-43de-96ea-d238df69afb0-kube-api-access-8zcgg\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.880991 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.880956    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a910569-1554-43de-96ea-d238df69afb0-isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.981446 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.981419    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcgg\" (UniqueName: \"kubernetes.io/projected/6a910569-1554-43de-96ea-d238df69afb0-kube-api-access-8zcgg\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.981604 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.981459    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a910569-1554-43de-96ea-d238df69afb0-isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.981604 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.981501    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a910569-1554-43de-96ea-d238df69afb0-proxy-tls\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.981604 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.981526    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a910569-1554-43de-96ea-d238df69afb0-kserve-provision-location\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.981970 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.981950    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a910569-1554-43de-96ea-d238df69afb0-kserve-provision-location\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.982214 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.982197    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a910569-1554-43de-96ea-d238df69afb0-isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.984050 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.984030    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a910569-1554-43de-96ea-d238df69afb0-proxy-tls\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:33.989239 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:33.989219    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcgg\" (UniqueName: \"kubernetes.io/projected/6a910569-1554-43de-96ea-d238df69afb0-kube-api-access-8zcgg\") pod \"isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") " pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:34.092931 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:34.092906    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:34.210921 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:34.210890    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"]
Apr 24 22:59:34.216550 ip-10-0-134-116 kubenswrapper[2543]: W0424 22:59:34.216506    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a910569_1554_43de_96ea_d238df69afb0.slice/crio-ed1c1635e093d5077ceaf47e6072783a7e4f251a03403322c959d610a0080821 WatchSource:0}: Error finding container ed1c1635e093d5077ceaf47e6072783a7e4f251a03403322c959d610a0080821: Status 404 returned error can't find the container with id ed1c1635e093d5077ceaf47e6072783a7e4f251a03403322c959d610a0080821
Apr 24 22:59:34.217897 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:34.217878    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 22:59:34.505481 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:34.505402    2543 generic.go:358] "Generic (PLEG): container finished" podID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerID="e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92" exitCode=2
Apr 24 22:59:34.505615 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:34.505486    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" event={"ID":"77e27b2b-a351-4e06-8d3e-2f881a955071","Type":"ContainerDied","Data":"e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92"}
Apr 24 22:59:34.506796 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:34.506773    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" event={"ID":"6a910569-1554-43de-96ea-d238df69afb0","Type":"ContainerStarted","Data":"d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b"}
Apr 24 22:59:34.506906 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:34.506806    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" event={"ID":"6a910569-1554-43de-96ea-d238df69afb0","Type":"ContainerStarted","Data":"ed1c1635e093d5077ceaf47e6072783a7e4f251a03403322c959d610a0080821"}
Apr 24 22:59:35.256217 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:35.256181    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Apr 24 22:59:35.260481 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:35.260451    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Apr 24 22:59:36.797185 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:36.797164    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:59:36.904031 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:36.903951    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e27b2b-a351-4e06-8d3e-2f881a955071-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"77e27b2b-a351-4e06-8d3e-2f881a955071\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") "
Apr 24 22:59:36.904031 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:36.904007    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e27b2b-a351-4e06-8d3e-2f881a955071-proxy-tls\") pod \"77e27b2b-a351-4e06-8d3e-2f881a955071\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") "
Apr 24 22:59:36.904031 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:36.904029    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtqvn\" (UniqueName: \"kubernetes.io/projected/77e27b2b-a351-4e06-8d3e-2f881a955071-kube-api-access-xtqvn\") pod \"77e27b2b-a351-4e06-8d3e-2f881a955071\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") "
Apr 24 22:59:36.904309 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:36.904085    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e27b2b-a351-4e06-8d3e-2f881a955071-kserve-provision-location\") pod \"77e27b2b-a351-4e06-8d3e-2f881a955071\" (UID: \"77e27b2b-a351-4e06-8d3e-2f881a955071\") "
Apr 24 22:59:36.904309 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:36.904258    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e27b2b-a351-4e06-8d3e-2f881a955071-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "77e27b2b-a351-4e06-8d3e-2f881a955071" (UID: "77e27b2b-a351-4e06-8d3e-2f881a955071"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 22:59:36.904452 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:36.904437    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e27b2b-a351-4e06-8d3e-2f881a955071-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77e27b2b-a351-4e06-8d3e-2f881a955071" (UID: "77e27b2b-a351-4e06-8d3e-2f881a955071"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 22:59:36.906142 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:36.906117    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e27b2b-a351-4e06-8d3e-2f881a955071-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77e27b2b-a351-4e06-8d3e-2f881a955071" (UID: "77e27b2b-a351-4e06-8d3e-2f881a955071"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 22:59:36.906257 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:36.906184    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e27b2b-a351-4e06-8d3e-2f881a955071-kube-api-access-xtqvn" (OuterVolumeSpecName: "kube-api-access-xtqvn") pod "77e27b2b-a351-4e06-8d3e-2f881a955071" (UID: "77e27b2b-a351-4e06-8d3e-2f881a955071"). InnerVolumeSpecName "kube-api-access-xtqvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 22:59:37.004559 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.004534    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77e27b2b-a351-4e06-8d3e-2f881a955071-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:59:37.004559 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.004557    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e27b2b-a351-4e06-8d3e-2f881a955071-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:59:37.004704 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.004566    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtqvn\" (UniqueName: \"kubernetes.io/projected/77e27b2b-a351-4e06-8d3e-2f881a955071-kube-api-access-xtqvn\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:59:37.004704 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.004577    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77e27b2b-a351-4e06-8d3e-2f881a955071-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 22:59:37.517567 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.517532    2543 generic.go:358] "Generic (PLEG): container finished" podID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerID="7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc" exitCode=0
Apr 24 22:59:37.517758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.517611    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" event={"ID":"77e27b2b-a351-4e06-8d3e-2f881a955071","Type":"ContainerDied","Data":"7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc"}
Apr 24 22:59:37.517758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.517652    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857" event={"ID":"77e27b2b-a351-4e06-8d3e-2f881a955071","Type":"ContainerDied","Data":"af62c5d04296bc3baddee4fd6dc78806d1a8298ad9721966bff0353a4473c70f"}
Apr 24 22:59:37.517758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.517654    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"
Apr 24 22:59:37.517758 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.517669    2543 scope.go:117] "RemoveContainer" containerID="e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92"
Apr 24 22:59:37.525403 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.525384    2543 scope.go:117] "RemoveContainer" containerID="7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc"
Apr 24 22:59:37.532078 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.532062    2543 scope.go:117] "RemoveContainer" containerID="38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a"
Apr 24 22:59:37.535738 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.535717    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"]
Apr 24 22:59:37.539066 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.539052    2543 scope.go:117] "RemoveContainer" containerID="e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92"
Apr 24 22:59:37.539322 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:59:37.539304    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92\": container with ID starting with e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92 not found: ID does not exist" containerID="e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92"
Apr 24 22:59:37.539543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.539359    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92"} err="failed to get container status \"e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92\": rpc error: code = NotFound desc = could not find container \"e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92\": container with ID starting with e064d6fe1f2533c5c3671047c0a81bd835c37bcee8e6a9745c2fc7f0b1dfcd92 not found: ID does not exist"
Apr 24 22:59:37.539543 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.539384    2543 scope.go:117] "RemoveContainer" containerID="7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc"
Apr 24 22:59:37.539706 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:59:37.539682    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc\": container with ID starting with 7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc not found: ID does not exist" containerID="7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc"
Apr 24 22:59:37.539753 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.539715    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc"} err="failed to get container status \"7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc\": rpc error: code = NotFound desc = could not find container \"7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc\": container with ID starting with 7311062f5701924cd01ffd0aa4c2a4605a66a7ac7dc343f4fcb5b96d2ccf32fc not found: ID does not exist"
Apr 24 22:59:37.539753 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.539737    2543 scope.go:117] "RemoveContainer" containerID="38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a"
Apr 24 22:59:37.540076 ip-10-0-134-116 kubenswrapper[2543]: E0424 22:59:37.539990    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a\": container with ID starting with 38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a not found: ID does not exist" containerID="38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a"
Apr 24 22:59:37.540076 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.540020    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a"} err="failed to get container status \"38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a\": rpc error: code = NotFound desc = could not find container \"38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a\": container with ID starting with 38a1eeed4ea6ac95f3695180913f72d35f43b28f7670697dabd078472454481a not found: ID does not exist"
Apr 24 22:59:37.541416 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:37.541398    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-bm857"]
Apr 24 22:59:38.522700 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:38.522668    2543 generic.go:358] "Generic (PLEG): container finished" podID="6a910569-1554-43de-96ea-d238df69afb0" containerID="d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b" exitCode=0
Apr 24 22:59:38.523059 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:38.522748    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" event={"ID":"6a910569-1554-43de-96ea-d238df69afb0","Type":"ContainerDied","Data":"d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b"}
Apr 24 22:59:39.054986 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:39.054954    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" path="/var/lib/kubelet/pods/77e27b2b-a351-4e06-8d3e-2f881a955071/volumes"
Apr 24 22:59:39.526827 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:39.526795    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" event={"ID":"6a910569-1554-43de-96ea-d238df69afb0","Type":"ContainerStarted","Data":"2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b"}
Apr 24 22:59:39.526827 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:39.526829    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" event={"ID":"6a910569-1554-43de-96ea-d238df69afb0","Type":"ContainerStarted","Data":"4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161"}
Apr 24 22:59:39.527197 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:39.527116    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:39.527235 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:39.527222    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:39.528456 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:39.528432    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 24 22:59:39.545234 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:39.545200    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podStartSLOduration=6.545188861 podStartE2EDuration="6.545188861s" podCreationTimestamp="2026-04-24 22:59:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 22:59:39.544193804 +0000 UTC m=+1782.970591348" watchObservedRunningTime="2026-04-24 22:59:39.545188861 +0000 UTC m=+1782.971586390"
Apr 24 22:59:40.530472 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:40.530431    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 24 22:59:45.535063 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:45.535037    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 22:59:45.535670 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:45.535646    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 24 22:59:55.536561 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:55.536520    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 24 22:59:57.489785 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:57.489761    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 22:59:57.492443 ip-10-0-134-116 kubenswrapper[2543]: I0424 22:59:57.492424    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:00:05.536212 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:05.536172    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 24 23:00:15.535844 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:15.535799    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 24 23:00:25.536102 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:25.536063    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 24 23:00:35.535848 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:35.535809    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 24 23:00:45.536122 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:45.536092    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 23:00:53.906155 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.906113    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"]
Apr 24 23:00:53.906519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.906415    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="storage-initializer"
Apr 24 23:00:53.906519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.906427    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="storage-initializer"
Apr 24 23:00:53.906519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.906439    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container"
Apr 24 23:00:53.906519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.906444    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container"
Apr 24 23:00:53.906519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.906455    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kube-rbac-proxy"
Apr 24 23:00:53.906519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.906460    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kube-rbac-proxy"
Apr 24 23:00:53.906519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.906504    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kube-rbac-proxy"
Apr 24 23:00:53.906519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.906513    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="77e27b2b-a351-4e06-8d3e-2f881a955071" containerName="kserve-container"
Apr 24 23:00:53.909389 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.909374    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:53.911878 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.911853    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 24 23:00:53.912027 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.912008    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-0c0f7e-predictor-serving-cert\""
Apr 24 23:00:53.912113 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.912014    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-0c0f7e\""
Apr 24 23:00:53.912185 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.912172    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-0c0f7e-dockercfg-hdsb9\""
Apr 24 23:00:53.912321 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.912289    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\""
Apr 24 23:00:53.922785 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.922765    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"]
Apr 24 23:00:53.967081 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.967052    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc668aa-c54d-4cce-8008-108e3b54aee5-kserve-provision-location\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:53.967188 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.967106    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-cabundle-cert\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:53.967188 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.967159    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg94b\" (UniqueName: \"kubernetes.io/projected/ebc668aa-c54d-4cce-8008-108e3b54aee5-kube-api-access-cg94b\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:53.967264 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.967194    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc668aa-c54d-4cce-8008-108e3b54aee5-proxy-tls\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:53.967264 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:53.967221    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.068461 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.068433    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc668aa-c54d-4cce-8008-108e3b54aee5-proxy-tls\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.068604 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.068469    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.068604 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:00:54.068575    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-serving-cert: secret "isvc-secondary-0c0f7e-predictor-serving-cert" not found
Apr 24 23:00:54.068721 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.068607    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc668aa-c54d-4cce-8008-108e3b54aee5-kserve-provision-location\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.068721 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:00:54.068644    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebc668aa-c54d-4cce-8008-108e3b54aee5-proxy-tls podName:ebc668aa-c54d-4cce-8008-108e3b54aee5 nodeName:}" failed. No retries permitted until 2026-04-24 23:00:54.568623111 +0000 UTC m=+1857.995020626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ebc668aa-c54d-4cce-8008-108e3b54aee5-proxy-tls") pod "isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw" (UID: "ebc668aa-c54d-4cce-8008-108e3b54aee5") : secret "isvc-secondary-0c0f7e-predictor-serving-cert" not found
Apr 24 23:00:54.068721 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.068701    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-cabundle-cert\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.068901 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.068737    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cg94b\" (UniqueName: \"kubernetes.io/projected/ebc668aa-c54d-4cce-8008-108e3b54aee5-kube-api-access-cg94b\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.068960 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.068933    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc668aa-c54d-4cce-8008-108e3b54aee5-kserve-provision-location\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.069214 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.069192    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.069297 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.069282    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-cabundle-cert\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.076791 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.076775    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg94b\" (UniqueName: \"kubernetes.io/projected/ebc668aa-c54d-4cce-8008-108e3b54aee5-kube-api-access-cg94b\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.571773 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.571746    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc668aa-c54d-4cce-8008-108e3b54aee5-proxy-tls\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.574255 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.574224    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc668aa-c54d-4cce-8008-108e3b54aee5-proxy-tls\") pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") " pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.819542 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.819508    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:00:54.940417 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:54.940392    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"]
Apr 24 23:00:54.942588 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:00:54.942559    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc668aa_c54d_4cce_8008_108e3b54aee5.slice/crio-2998507605d569f4d7b1fa07f6ee9f0ab4c7031a95ab6149efc50ab66097e17d WatchSource:0}: Error finding container 2998507605d569f4d7b1fa07f6ee9f0ab4c7031a95ab6149efc50ab66097e17d: Status 404 returned error can't find the container with id 2998507605d569f4d7b1fa07f6ee9f0ab4c7031a95ab6149efc50ab66097e17d
Apr 24 23:00:55.740088 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:55.740048    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw" event={"ID":"ebc668aa-c54d-4cce-8008-108e3b54aee5","Type":"ContainerStarted","Data":"31deb4929c3bc8cb0b9a5971f6862a16105cf553047def84bea987f426cb78ea"}
Apr 24 23:00:55.740088 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:00:55.740091    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw" event={"ID":"ebc668aa-c54d-4cce-8008-108e3b54aee5","Type":"ContainerStarted","Data":"2998507605d569f4d7b1fa07f6ee9f0ab4c7031a95ab6149efc50ab66097e17d"}
Apr 24 23:01:01.759130 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:01.759101    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_ebc668aa-c54d-4cce-8008-108e3b54aee5/storage-initializer/0.log"
Apr 24 23:01:01.759555 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:01.759141    2543 generic.go:358] "Generic (PLEG): container finished" podID="ebc668aa-c54d-4cce-8008-108e3b54aee5" containerID="31deb4929c3bc8cb0b9a5971f6862a16105cf553047def84bea987f426cb78ea" exitCode=1
Apr 24 23:01:01.759555 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:01.759171    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw" event={"ID":"ebc668aa-c54d-4cce-8008-108e3b54aee5","Type":"ContainerDied","Data":"31deb4929c3bc8cb0b9a5971f6862a16105cf553047def84bea987f426cb78ea"}
Apr 24 23:01:02.765380 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:02.765326    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_ebc668aa-c54d-4cce-8008-108e3b54aee5/storage-initializer/0.log"
Apr 24 23:01:02.765742 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:02.765451    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw" event={"ID":"ebc668aa-c54d-4cce-8008-108e3b54aee5","Type":"ContainerStarted","Data":"1ce104d648044e66d5eba7b7d869af06bd3c7ea9bc02280db5298066871d721c"}
Apr 24 23:01:05.775902 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:05.775876    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_ebc668aa-c54d-4cce-8008-108e3b54aee5/storage-initializer/1.log"
Apr 24 23:01:05.776264 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:05.776247    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_ebc668aa-c54d-4cce-8008-108e3b54aee5/storage-initializer/0.log"
Apr 24 23:01:05.776311 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:05.776281    2543 generic.go:358] "Generic (PLEG): container finished" podID="ebc668aa-c54d-4cce-8008-108e3b54aee5" containerID="1ce104d648044e66d5eba7b7d869af06bd3c7ea9bc02280db5298066871d721c" exitCode=1
Apr 24 23:01:05.776378 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:05.776365    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw" event={"ID":"ebc668aa-c54d-4cce-8008-108e3b54aee5","Type":"ContainerDied","Data":"1ce104d648044e66d5eba7b7d869af06bd3c7ea9bc02280db5298066871d721c"}
Apr 24 23:01:05.776437 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:05.776403    2543 scope.go:117] "RemoveContainer" containerID="31deb4929c3bc8cb0b9a5971f6862a16105cf553047def84bea987f426cb78ea"
Apr 24 23:01:05.776771 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:05.776750    2543 scope.go:117] "RemoveContainer" containerID="31deb4929c3bc8cb0b9a5971f6862a16105cf553047def84bea987f426cb78ea"
Apr 24 23:01:05.786419 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:05.786392    2543 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_kserve-ci-e2e-test_ebc668aa-c54d-4cce-8008-108e3b54aee5_0 in pod sandbox 2998507605d569f4d7b1fa07f6ee9f0ab4c7031a95ab6149efc50ab66097e17d from index: no such id: '31deb4929c3bc8cb0b9a5971f6862a16105cf553047def84bea987f426cb78ea'" containerID="31deb4929c3bc8cb0b9a5971f6862a16105cf553047def84bea987f426cb78ea"
Apr 24 23:01:05.786489 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:05.786444    2543 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_kserve-ci-e2e-test_ebc668aa-c54d-4cce-8008-108e3b54aee5_0 in pod sandbox 2998507605d569f4d7b1fa07f6ee9f0ab4c7031a95ab6149efc50ab66097e17d from index: no such id: '31deb4929c3bc8cb0b9a5971f6862a16105cf553047def84bea987f426cb78ea'; Skipping pod \"isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_kserve-ci-e2e-test(ebc668aa-c54d-4cce-8008-108e3b54aee5)\"" logger="UnhandledError"
Apr 24 23:01:05.787744 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:05.787725    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_kserve-ci-e2e-test(ebc668aa-c54d-4cce-8008-108e3b54aee5)\"" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw" podUID="ebc668aa-c54d-4cce-8008-108e3b54aee5"
Apr 24 23:01:06.780378 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:06.780300    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_ebc668aa-c54d-4cce-8008-108e3b54aee5/storage-initializer/1.log"
Apr 24 23:01:13.961524 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:13.961483    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"]
Apr 24 23:01:14.010168 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.010137    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"]
Apr 24 23:01:14.011207 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.011142    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" containerID="cri-o://4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161" gracePeriod=30
Apr 24 23:01:14.012414 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.011398    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kube-rbac-proxy" containerID="cri-o://2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b" gracePeriod=30
Apr 24 23:01:14.074073 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.074035    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"]
Apr 24 23:01:14.078556 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.078534    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.081136 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.081103    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c1d19b-predictor-serving-cert\""
Apr 24 23:01:14.081136 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.081119    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\""
Apr 24 23:01:14.081314 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.081144    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-c1d19b\""
Apr 24 23:01:14.081314 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.081119    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-c1d19b-dockercfg-b8qpg\""
Apr 24 23:01:14.084972 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.084953    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"]
Apr 24 23:01:14.137008 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.136986    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_ebc668aa-c54d-4cce-8008-108e3b54aee5/storage-initializer/1.log"
Apr 24 23:01:14.137111 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.137058    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:01:14.220739 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.220666    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc668aa-c54d-4cce-8008-108e3b54aee5-kserve-provision-location\") pod \"ebc668aa-c54d-4cce-8008-108e3b54aee5\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") "
Apr 24 23:01:14.220739 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.220711    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\") pod \"ebc668aa-c54d-4cce-8008-108e3b54aee5\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") "
Apr 24 23:01:14.220914 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.220753    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-cabundle-cert\") pod \"ebc668aa-c54d-4cce-8008-108e3b54aee5\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") "
Apr 24 23:01:14.220914 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.220792    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc668aa-c54d-4cce-8008-108e3b54aee5-proxy-tls\") pod \"ebc668aa-c54d-4cce-8008-108e3b54aee5\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") "
Apr 24 23:01:14.220914 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.220820    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg94b\" (UniqueName: \"kubernetes.io/projected/ebc668aa-c54d-4cce-8008-108e3b54aee5-kube-api-access-cg94b\") pod \"ebc668aa-c54d-4cce-8008-108e3b54aee5\" (UID: \"ebc668aa-c54d-4cce-8008-108e3b54aee5\") "
Apr 24 23:01:14.220914 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.220900    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kserve-provision-location\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.221083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.220951    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvqld\" (UniqueName: \"kubernetes.io/projected/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kube-api-access-kvqld\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.221083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.220979    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-cabundle-cert\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.221083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.220979    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc668aa-c54d-4cce-8008-108e3b54aee5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebc668aa-c54d-4cce-8008-108e3b54aee5" (UID: "ebc668aa-c54d-4cce-8008-108e3b54aee5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:01:14.221216 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.221121    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ebc668aa-c54d-4cce-8008-108e3b54aee5" (UID: "ebc668aa-c54d-4cce-8008-108e3b54aee5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:01:14.221216 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.221127    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.221216 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.221135    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config") pod "ebc668aa-c54d-4cce-8008-108e3b54aee5" (UID: "ebc668aa-c54d-4cce-8008-108e3b54aee5"). InnerVolumeSpecName "isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:01:14.221216 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.221197    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-proxy-tls\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.221399 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.221254    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-cabundle-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:14.221399 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.221268    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc668aa-c54d-4cce-8008-108e3b54aee5-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:14.221399 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.221284    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebc668aa-c54d-4cce-8008-108e3b54aee5-isvc-secondary-0c0f7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:14.222875 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.222853    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc668aa-c54d-4cce-8008-108e3b54aee5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebc668aa-c54d-4cce-8008-108e3b54aee5" (UID: "ebc668aa-c54d-4cce-8008-108e3b54aee5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:01:14.222995 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.222880    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc668aa-c54d-4cce-8008-108e3b54aee5-kube-api-access-cg94b" (OuterVolumeSpecName: "kube-api-access-cg94b") pod "ebc668aa-c54d-4cce-8008-108e3b54aee5" (UID: "ebc668aa-c54d-4cce-8008-108e3b54aee5"). InnerVolumeSpecName "kube-api-access-cg94b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:01:14.322489 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.322459    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.322643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.322501    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-proxy-tls\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.322643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.322531    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kserve-provision-location\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.322643 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:14.322628    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-serving-cert: secret "isvc-init-fail-c1d19b-predictor-serving-cert" not found
Apr 24 23:01:14.322755 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:14.322693    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-proxy-tls podName:c1565c3d-5003-4b53-b44e-3fafdd1e1d0b nodeName:}" failed. No retries permitted until 2026-04-24 23:01:14.82267346 +0000 UTC m=+1878.249070982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-proxy-tls") pod "isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc" (UID: "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b") : secret "isvc-init-fail-c1d19b-predictor-serving-cert" not found
Apr 24 23:01:14.322755 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.322715    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvqld\" (UniqueName: \"kubernetes.io/projected/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kube-api-access-kvqld\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.322852 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.322751    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-cabundle-cert\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.322852 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.322815    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebc668aa-c54d-4cce-8008-108e3b54aee5-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:14.322852 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.322830    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg94b\" (UniqueName: \"kubernetes.io/projected/ebc668aa-c54d-4cce-8008-108e3b54aee5-kube-api-access-cg94b\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:14.322957 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.322878    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kserve-provision-location\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.323212 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.323193    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.323309 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.323293    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-cabundle-cert\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.330827 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.330810    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvqld\" (UniqueName: \"kubernetes.io/projected/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kube-api-access-kvqld\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.801968 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.801939    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw_ebc668aa-c54d-4cce-8008-108e3b54aee5/storage-initializer/1.log"
Apr 24 23:01:14.802115 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.802046    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw" event={"ID":"ebc668aa-c54d-4cce-8008-108e3b54aee5","Type":"ContainerDied","Data":"2998507605d569f4d7b1fa07f6ee9f0ab4c7031a95ab6149efc50ab66097e17d"}
Apr 24 23:01:14.802115 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.802059    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"
Apr 24 23:01:14.802115 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.802075    2543 scope.go:117] "RemoveContainer" containerID="1ce104d648044e66d5eba7b7d869af06bd3c7ea9bc02280db5298066871d721c"
Apr 24 23:01:14.804190 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.804168    2543 generic.go:358] "Generic (PLEG): container finished" podID="6a910569-1554-43de-96ea-d238df69afb0" containerID="2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b" exitCode=2
Apr 24 23:01:14.804308 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.804213    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" event={"ID":"6a910569-1554-43de-96ea-d238df69afb0","Type":"ContainerDied","Data":"2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b"}
Apr 24 23:01:14.826719 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.826698    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-proxy-tls\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.829554 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.829530    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-proxy-tls\") pod \"isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") " pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:14.837653 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.837623    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"]
Apr 24 23:01:14.840461 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.840442    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0c0f7e-predictor-6bfc5d8786-ht5fw"]
Apr 24 23:01:14.989502 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:14.989472    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:15.057055 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:15.056980    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc668aa-c54d-4cce-8008-108e3b54aee5" path="/var/lib/kubelet/pods/ebc668aa-c54d-4cce-8008-108e3b54aee5/volumes"
Apr 24 23:01:15.113012 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:15.112987    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"]
Apr 24 23:01:15.114863 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:01:15.114831    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1565c3d_5003_4b53_b44e_3fafdd1e1d0b.slice/crio-e2ec4172ca18252f0d9e0a1de2c016ecf562af58ae3950bc20bbf3849c78dabd WatchSource:0}: Error finding container e2ec4172ca18252f0d9e0a1de2c016ecf562af58ae3950bc20bbf3849c78dabd: Status 404 returned error can't find the container with id e2ec4172ca18252f0d9e0a1de2c016ecf562af58ae3950bc20bbf3849c78dabd
Apr 24 23:01:15.531323 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:15.531283    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Apr 24 23:01:15.535576 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:15.535554    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Apr 24 23:01:15.809120 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:15.809036    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc" event={"ID":"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b","Type":"ContainerStarted","Data":"13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4"}
Apr 24 23:01:15.809120 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:15.809082    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc" event={"ID":"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b","Type":"ContainerStarted","Data":"e2ec4172ca18252f0d9e0a1de2c016ecf562af58ae3950bc20bbf3849c78dabd"}
Apr 24 23:01:17.757524 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.757503    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 23:01:17.818150 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.818120    2543 generic.go:358] "Generic (PLEG): container finished" podID="6a910569-1554-43de-96ea-d238df69afb0" containerID="4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161" exitCode=0
Apr 24 23:01:17.818271 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.818166    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" event={"ID":"6a910569-1554-43de-96ea-d238df69afb0","Type":"ContainerDied","Data":"4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161"}
Apr 24 23:01:17.818271 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.818198    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"
Apr 24 23:01:17.818271 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.818205    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj" event={"ID":"6a910569-1554-43de-96ea-d238df69afb0","Type":"ContainerDied","Data":"ed1c1635e093d5077ceaf47e6072783a7e4f251a03403322c959d610a0080821"}
Apr 24 23:01:17.818271 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.818221    2543 scope.go:117] "RemoveContainer" containerID="2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b"
Apr 24 23:01:17.825549 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.825531    2543 scope.go:117] "RemoveContainer" containerID="4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161"
Apr 24 23:01:17.832222 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.832207    2543 scope.go:117] "RemoveContainer" containerID="d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b"
Apr 24 23:01:17.838771 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.838756    2543 scope.go:117] "RemoveContainer" containerID="2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b"
Apr 24 23:01:17.839005 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:17.838988    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b\": container with ID starting with 2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b not found: ID does not exist" containerID="2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b"
Apr 24 23:01:17.839072 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.839013    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b"} err="failed to get container status \"2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b\": rpc error: code = NotFound desc = could not find container \"2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b\": container with ID starting with 2e8f8539f6c9e1bf2d5d59d992b78478475c024e7d463301541ff91b8c9c888b not found: ID does not exist"
Apr 24 23:01:17.839072 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.839029    2543 scope.go:117] "RemoveContainer" containerID="4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161"
Apr 24 23:01:17.839265 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:17.839245    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161\": container with ID starting with 4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161 not found: ID does not exist" containerID="4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161"
Apr 24 23:01:17.839309 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.839273    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161"} err="failed to get container status \"4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161\": rpc error: code = NotFound desc = could not find container \"4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161\": container with ID starting with 4aa52ef98edc30b15b32f044a7e4604a89339ec28afefdaf54d9de1795b1c161 not found: ID does not exist"
Apr 24 23:01:17.839309 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.839295    2543 scope.go:117] "RemoveContainer" containerID="d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b"
Apr 24 23:01:17.839582 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:17.839565    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b\": container with ID starting with d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b not found: ID does not exist" containerID="d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b"
Apr 24 23:01:17.839629 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.839587    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b"} err="failed to get container status \"d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b\": rpc error: code = NotFound desc = could not find container \"d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b\": container with ID starting with d45d72e406705c3a8b22bb8bf0717ea7876d1a5d891f378a34e8957da2f8bc7b not found: ID does not exist"
Apr 24 23:01:17.850219 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.850206    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a910569-1554-43de-96ea-d238df69afb0-kserve-provision-location\") pod \"6a910569-1554-43de-96ea-d238df69afb0\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") "
Apr 24 23:01:17.850269 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.850251    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a910569-1554-43de-96ea-d238df69afb0-isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\") pod \"6a910569-1554-43de-96ea-d238df69afb0\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") "
Apr 24 23:01:17.850305 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.850291    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a910569-1554-43de-96ea-d238df69afb0-proxy-tls\") pod \"6a910569-1554-43de-96ea-d238df69afb0\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") "
Apr 24 23:01:17.850356 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.850307    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zcgg\" (UniqueName: \"kubernetes.io/projected/6a910569-1554-43de-96ea-d238df69afb0-kube-api-access-8zcgg\") pod \"6a910569-1554-43de-96ea-d238df69afb0\" (UID: \"6a910569-1554-43de-96ea-d238df69afb0\") "
Apr 24 23:01:17.850540 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.850517    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a910569-1554-43de-96ea-d238df69afb0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a910569-1554-43de-96ea-d238df69afb0" (UID: "6a910569-1554-43de-96ea-d238df69afb0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:01:17.850637 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.850619    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a910569-1554-43de-96ea-d238df69afb0-isvc-primary-0c0f7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-0c0f7e-kube-rbac-proxy-sar-config") pod "6a910569-1554-43de-96ea-d238df69afb0" (UID: "6a910569-1554-43de-96ea-d238df69afb0"). InnerVolumeSpecName "isvc-primary-0c0f7e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:01:17.852247 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.852226    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a910569-1554-43de-96ea-d238df69afb0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a910569-1554-43de-96ea-d238df69afb0" (UID: "6a910569-1554-43de-96ea-d238df69afb0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:01:17.852390 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.852369    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a910569-1554-43de-96ea-d238df69afb0-kube-api-access-8zcgg" (OuterVolumeSpecName: "kube-api-access-8zcgg") pod "6a910569-1554-43de-96ea-d238df69afb0" (UID: "6a910569-1554-43de-96ea-d238df69afb0"). InnerVolumeSpecName "kube-api-access-8zcgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:01:17.951462 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.951400    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a910569-1554-43de-96ea-d238df69afb0-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:17.951462 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.951432    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a910569-1554-43de-96ea-d238df69afb0-isvc-primary-0c0f7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:17.951462 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.951447    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a910569-1554-43de-96ea-d238df69afb0-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:17.951462 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:17.951461    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zcgg\" (UniqueName: \"kubernetes.io/projected/6a910569-1554-43de-96ea-d238df69afb0-kube-api-access-8zcgg\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:18.139458 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:18.139432    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"]
Apr 24 23:01:18.143745 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:18.143723    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0c0f7e-predictor-66b6bd9685-cfmfj"]
Apr 24 23:01:19.054691 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:19.054660    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a910569-1554-43de-96ea-d238df69afb0" path="/var/lib/kubelet/pods/6a910569-1554-43de-96ea-d238df69afb0/volumes"
Apr 24 23:01:20.828505 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:20.828478    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc_c1565c3d-5003-4b53-b44e-3fafdd1e1d0b/storage-initializer/0.log"
Apr 24 23:01:20.828792 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:20.828521    2543 generic.go:358] "Generic (PLEG): container finished" podID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" containerID="13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4" exitCode=1
Apr 24 23:01:20.828792 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:20.828555    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc" event={"ID":"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b","Type":"ContainerDied","Data":"13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4"}
Apr 24 23:01:21.832665 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:21.832636    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc_c1565c3d-5003-4b53-b44e-3fafdd1e1d0b/storage-initializer/0.log"
Apr 24 23:01:21.833017 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:21.832729    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc" event={"ID":"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b","Type":"ContainerStarted","Data":"bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9"}
Apr 24 23:01:24.085468 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.085438    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"]
Apr 24 23:01:24.085921 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.085730    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc" podUID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" containerName="storage-initializer" containerID="cri-o://bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9" gracePeriod=30
Apr 24 23:01:24.213999 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.213966    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"]
Apr 24 23:01:24.214279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214266    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kube-rbac-proxy"
Apr 24 23:01:24.214319 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214281    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kube-rbac-proxy"
Apr 24 23:01:24.214319 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214293    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container"
Apr 24 23:01:24.214319 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214299    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container"
Apr 24 23:01:24.214319 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214305    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc668aa-c54d-4cce-8008-108e3b54aee5" containerName="storage-initializer"
Apr 24 23:01:24.214319 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214310    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc668aa-c54d-4cce-8008-108e3b54aee5" containerName="storage-initializer"
Apr 24 23:01:24.214319 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214318    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="storage-initializer"
Apr 24 23:01:24.214513 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214325    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="storage-initializer"
Apr 24 23:01:24.214513 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214405    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kserve-container"
Apr 24 23:01:24.214513 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214416    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="ebc668aa-c54d-4cce-8008-108e3b54aee5" containerName="storage-initializer"
Apr 24 23:01:24.214513 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214425    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="6a910569-1554-43de-96ea-d238df69afb0" containerName="kube-rbac-proxy"
Apr 24 23:01:24.214513 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214474    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc668aa-c54d-4cce-8008-108e3b54aee5" containerName="storage-initializer"
Apr 24 23:01:24.214513 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214480    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc668aa-c54d-4cce-8008-108e3b54aee5" containerName="storage-initializer"
Apr 24 23:01:24.214682 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.214536    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="ebc668aa-c54d-4cce-8008-108e3b54aee5" containerName="storage-initializer"
Apr 24 23:01:24.217439 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.217424    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.220280 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.220262    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-hkqwb\""
Apr 24 23:01:24.220442 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.220267    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Apr 24 23:01:24.221075 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.221055    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Apr 24 23:01:24.229568 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.229548    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"]
Apr 24 23:01:24.298458 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.298438    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596hm\" (UniqueName: \"kubernetes.io/projected/b0875d60-67cb-4bc1-afad-ee12982d211e-kube-api-access-596hm\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.298565 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.298472    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0875d60-67cb-4bc1-afad-ee12982d211e-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.298565 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.298504    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0875d60-67cb-4bc1-afad-ee12982d211e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.298681 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.298567    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0875d60-67cb-4bc1-afad-ee12982d211e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.399207 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.399150    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-596hm\" (UniqueName: \"kubernetes.io/projected/b0875d60-67cb-4bc1-afad-ee12982d211e-kube-api-access-596hm\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.399207 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.399186    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0875d60-67cb-4bc1-afad-ee12982d211e-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.399390 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.399234    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0875d60-67cb-4bc1-afad-ee12982d211e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.399390 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.399267    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0875d60-67cb-4bc1-afad-ee12982d211e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.399678 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.399660    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0875d60-67cb-4bc1-afad-ee12982d211e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.399941 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.399921    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0875d60-67cb-4bc1-afad-ee12982d211e-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.401717 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.401701    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0875d60-67cb-4bc1-afad-ee12982d211e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.407699 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.407680    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-596hm\" (UniqueName: \"kubernetes.io/projected/b0875d60-67cb-4bc1-afad-ee12982d211e-kube-api-access-596hm\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.527524 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.527501    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:24.641610 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.641589    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"]
Apr 24 23:01:24.644051 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:01:24.644022    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0875d60_67cb_4bc1_afad_ee12982d211e.slice/crio-0cbaea4c7682bf83788da6373887e06987c54e068cef266e6087e775e4883afd WatchSource:0}: Error finding container 0cbaea4c7682bf83788da6373887e06987c54e068cef266e6087e775e4883afd: Status 404 returned error can't find the container with id 0cbaea4c7682bf83788da6373887e06987c54e068cef266e6087e775e4883afd
Apr 24 23:01:24.842363 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.842305    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" event={"ID":"b0875d60-67cb-4bc1-afad-ee12982d211e","Type":"ContainerStarted","Data":"133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e"}
Apr 24 23:01:24.842527 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:24.842368    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" event={"ID":"b0875d60-67cb-4bc1-afad-ee12982d211e","Type":"ContainerStarted","Data":"0cbaea4c7682bf83788da6373887e06987c54e068cef266e6087e775e4883afd"}
Apr 24 23:01:26.626119 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.626095    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc_c1565c3d-5003-4b53-b44e-3fafdd1e1d0b/storage-initializer/1.log"
Apr 24 23:01:26.626498 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.626483    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc_c1565c3d-5003-4b53-b44e-3fafdd1e1d0b/storage-initializer/0.log"
Apr 24 23:01:26.626558 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.626548    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:26.717283 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.717260    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\") pod \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") "
Apr 24 23:01:26.717440 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.717304    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-cabundle-cert\") pod \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") "
Apr 24 23:01:26.717440 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.717334    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kserve-provision-location\") pod \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") "
Apr 24 23:01:26.717440 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.717386    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-proxy-tls\") pod \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") "
Apr 24 23:01:26.717440 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.717422    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvqld\" (UniqueName: \"kubernetes.io/projected/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kube-api-access-kvqld\") pod \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\" (UID: \"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b\") "
Apr 24 23:01:26.717668 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.717643    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" (UID: "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:01:26.717726 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.717679    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config") pod "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" (UID: "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b"). InnerVolumeSpecName "isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:01:26.717726 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.717692    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" (UID: "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:01:26.719614 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.719593    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kube-api-access-kvqld" (OuterVolumeSpecName: "kube-api-access-kvqld") pod "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" (UID: "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b"). InnerVolumeSpecName "kube-api-access-kvqld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:01:26.719673 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.719651    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" (UID: "c1565c3d-5003-4b53-b44e-3fafdd1e1d0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:01:26.818741 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.818719    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:26.818741 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.818742    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvqld\" (UniqueName: \"kubernetes.io/projected/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kube-api-access-kvqld\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:26.818858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.818751    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-isvc-init-fail-c1d19b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:26.818858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.818763    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-cabundle-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:26.818858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.818771    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:01:26.849070 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.849051    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc_c1565c3d-5003-4b53-b44e-3fafdd1e1d0b/storage-initializer/1.log"
Apr 24 23:01:26.849423 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.849404    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc_c1565c3d-5003-4b53-b44e-3fafdd1e1d0b/storage-initializer/0.log"
Apr 24 23:01:26.849515 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.849442    2543 generic.go:358] "Generic (PLEG): container finished" podID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" containerID="bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9" exitCode=1
Apr 24 23:01:26.849515 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.849510    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"
Apr 24 23:01:26.849598 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.849535    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc" event={"ID":"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b","Type":"ContainerDied","Data":"bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9"}
Apr 24 23:01:26.849598 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.849570    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc" event={"ID":"c1565c3d-5003-4b53-b44e-3fafdd1e1d0b","Type":"ContainerDied","Data":"e2ec4172ca18252f0d9e0a1de2c016ecf562af58ae3950bc20bbf3849c78dabd"}
Apr 24 23:01:26.849598 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.849585    2543 scope.go:117] "RemoveContainer" containerID="bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9"
Apr 24 23:01:26.857326 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.857290    2543 scope.go:117] "RemoveContainer" containerID="13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4"
Apr 24 23:01:26.866755 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.866738    2543 scope.go:117] "RemoveContainer" containerID="bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9"
Apr 24 23:01:26.867009 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:26.866991    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9\": container with ID starting with bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9 not found: ID does not exist" containerID="bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9"
Apr 24 23:01:26.867067 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.867014    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9"} err="failed to get container status \"bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9\": rpc error: code = NotFound desc = could not find container \"bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9\": container with ID starting with bc449737b40906295d46267c5eb9df021a30c104363aadd27b2463b05f6e4ff9 not found: ID does not exist"
Apr 24 23:01:26.867067 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.867030    2543 scope.go:117] "RemoveContainer" containerID="13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4"
Apr 24 23:01:26.867253 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:01:26.867239    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4\": container with ID starting with 13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4 not found: ID does not exist" containerID="13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4"
Apr 24 23:01:26.867293 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.867256    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4"} err="failed to get container status \"13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4\": rpc error: code = NotFound desc = could not find container \"13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4\": container with ID starting with 13ddc0e6f71a55608dd75b9c112aa6be0f1a1d9f361655259c0d770c8fa23bb4 not found: ID does not exist"
Apr 24 23:01:26.888475 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.888453    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"]
Apr 24 23:01:26.892464 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:26.892444    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c1d19b-predictor-76c7889f6-mfqwc"]
Apr 24 23:01:27.059284 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:27.055901    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" path="/var/lib/kubelet/pods/c1565c3d-5003-4b53-b44e-3fafdd1e1d0b/volumes"
Apr 24 23:01:28.856489 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:28.856456    2543 generic.go:358] "Generic (PLEG): container finished" podID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerID="133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e" exitCode=0
Apr 24 23:01:28.856851 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:28.856517    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" event={"ID":"b0875d60-67cb-4bc1-afad-ee12982d211e","Type":"ContainerDied","Data":"133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e"}
Apr 24 23:01:58.953946 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:58.953907    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" event={"ID":"b0875d60-67cb-4bc1-afad-ee12982d211e","Type":"ContainerStarted","Data":"1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c"}
Apr 24 23:01:58.954422 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:58.953956    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" event={"ID":"b0875d60-67cb-4bc1-afad-ee12982d211e","Type":"ContainerStarted","Data":"39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450"}
Apr 24 23:01:58.954422 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:58.954255    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:58.954422 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:58.954380    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:01:58.955511 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:58.955484    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 24 23:01:58.973363 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:58.973311    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podStartSLOduration=5.449639853 podStartE2EDuration="34.973299655s" podCreationTimestamp="2026-04-24 23:01:24 +0000 UTC" firstStartedPulling="2026-04-24 23:01:28.857641283 +0000 UTC m=+1892.284038790" lastFinishedPulling="2026-04-24 23:01:58.381301082 +0000 UTC m=+1921.807698592" observedRunningTime="2026-04-24 23:01:58.971128115 +0000 UTC m=+1922.397525655" watchObservedRunningTime="2026-04-24 23:01:58.973299655 +0000 UTC m=+1922.399697184"
Apr 24 23:01:59.957459 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:01:59.957417    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 24 23:02:04.962738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:02:04.962707    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:02:04.963237 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:02:04.963212    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 24 23:02:14.963365 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:02:14.963306    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 24 23:02:24.963586 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:02:24.963548    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 24 23:02:34.963462 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:02:34.963420    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 24 23:02:44.963685 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:02:44.963597    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 24 23:02:54.963973 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:02:54.963934    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 24 23:03:04.963551 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:04.963510    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Apr 24 23:03:14.052265 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:14.052240    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:03:24.348673 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.348591    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"]
Apr 24 23:03:24.349110 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.349010    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container" containerID="cri-o://39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450" gracePeriod=30
Apr 24 23:03:24.349308 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.349186    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kube-rbac-proxy" containerID="cri-o://1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c" gracePeriod=30
Apr 24 23:03:24.432949 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.432905    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"]
Apr 24 23:03:24.433213 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.433201    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" containerName="storage-initializer"
Apr 24 23:03:24.433260 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.433214    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" containerName="storage-initializer"
Apr 24 23:03:24.433260 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.433226    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" containerName="storage-initializer"
Apr 24 23:03:24.433260 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.433233    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" containerName="storage-initializer"
Apr 24 23:03:24.433372 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.433281    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" containerName="storage-initializer"
Apr 24 23:03:24.433372 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.433291    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1565c3d-5003-4b53-b44e-3fafdd1e1d0b" containerName="storage-initializer"
Apr 24 23:03:24.436365 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.436329    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.438586 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.438564    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Apr 24 23:03:24.438733 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.438716    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Apr 24 23:03:24.445485 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.445462    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"]
Apr 24 23:03:24.488235 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.488209    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11cd5412-257f-43ce-adad-6006cb9e2c12-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.488366 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.488256    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqwmg\" (UniqueName: \"kubernetes.io/projected/11cd5412-257f-43ce-adad-6006cb9e2c12-kube-api-access-jqwmg\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.488366 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.488301    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11cd5412-257f-43ce-adad-6006cb9e2c12-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.488366 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.488319    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11cd5412-257f-43ce-adad-6006cb9e2c12-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.589649 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.589610    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11cd5412-257f-43ce-adad-6006cb9e2c12-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.589844 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.589664    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jqwmg\" (UniqueName: \"kubernetes.io/projected/11cd5412-257f-43ce-adad-6006cb9e2c12-kube-api-access-jqwmg\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.589844 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.589690    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11cd5412-257f-43ce-adad-6006cb9e2c12-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.589844 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.589708    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11cd5412-257f-43ce-adad-6006cb9e2c12-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.589844 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:03:24.589797    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Apr 24 23:03:24.590058 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:03:24.589870    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11cd5412-257f-43ce-adad-6006cb9e2c12-proxy-tls podName:11cd5412-257f-43ce-adad-6006cb9e2c12 nodeName:}" failed. No retries permitted until 2026-04-24 23:03:25.089848275 +0000 UTC m=+2008.516245789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/11cd5412-257f-43ce-adad-6006cb9e2c12-proxy-tls") pod "isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" (UID: "11cd5412-257f-43ce-adad-6006cb9e2c12") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Apr 24 23:03:24.590117 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.590066    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11cd5412-257f-43ce-adad-6006cb9e2c12-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.590276 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.590259    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11cd5412-257f-43ce-adad-6006cb9e2c12-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.598386 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.598362    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqwmg\" (UniqueName: \"kubernetes.io/projected/11cd5412-257f-43ce-adad-6006cb9e2c12-kube-api-access-jqwmg\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:24.958073 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:24.958033    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Apr 24 23:03:25.094487 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:25.094449    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11cd5412-257f-43ce-adad-6006cb9e2c12-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:25.096923 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:25.096899    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11cd5412-257f-43ce-adad-6006cb9e2c12-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:25.198079 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:25.198041    2543 generic.go:358] "Generic (PLEG): container finished" podID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerID="1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c" exitCode=2
Apr 24 23:03:25.198215 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:25.198116    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" event={"ID":"b0875d60-67cb-4bc1-afad-ee12982d211e","Type":"ContainerDied","Data":"1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c"}
Apr 24 23:03:25.346582 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:25.346473    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:25.464518 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:25.464484    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"]
Apr 24 23:03:25.469024 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:03:25.468992    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cd5412_257f_43ce_adad_6006cb9e2c12.slice/crio-3c65d7998f0587b18e115cce2381002dc58698f863540cb5a3ead61ca8b36927 WatchSource:0}: Error finding container 3c65d7998f0587b18e115cce2381002dc58698f863540cb5a3ead61ca8b36927: Status 404 returned error can't find the container with id 3c65d7998f0587b18e115cce2381002dc58698f863540cb5a3ead61ca8b36927
Apr 24 23:03:26.202157 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:26.202115    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" event={"ID":"11cd5412-257f-43ce-adad-6006cb9e2c12","Type":"ContainerStarted","Data":"e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b"}
Apr 24 23:03:26.202157 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:26.202152    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" event={"ID":"11cd5412-257f-43ce-adad-6006cb9e2c12","Type":"ContainerStarted","Data":"3c65d7998f0587b18e115cce2381002dc58698f863540cb5a3ead61ca8b36927"}
Apr 24 23:03:28.585516 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.585495    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:03:28.620686 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.620660    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0875d60-67cb-4bc1-afad-ee12982d211e-proxy-tls\") pod \"b0875d60-67cb-4bc1-afad-ee12982d211e\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") "
Apr 24 23:03:28.620805 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.620698    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0875d60-67cb-4bc1-afad-ee12982d211e-kserve-provision-location\") pod \"b0875d60-67cb-4bc1-afad-ee12982d211e\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") "
Apr 24 23:03:28.620805 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.620744    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0875d60-67cb-4bc1-afad-ee12982d211e-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"b0875d60-67cb-4bc1-afad-ee12982d211e\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") "
Apr 24 23:03:28.620805 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.620788    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-596hm\" (UniqueName: \"kubernetes.io/projected/b0875d60-67cb-4bc1-afad-ee12982d211e-kube-api-access-596hm\") pod \"b0875d60-67cb-4bc1-afad-ee12982d211e\" (UID: \"b0875d60-67cb-4bc1-afad-ee12982d211e\") "
Apr 24 23:03:28.621075 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.621051    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0875d60-67cb-4bc1-afad-ee12982d211e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0875d60-67cb-4bc1-afad-ee12982d211e" (UID: "b0875d60-67cb-4bc1-afad-ee12982d211e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:03:28.621138 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.621084    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0875d60-67cb-4bc1-afad-ee12982d211e-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "b0875d60-67cb-4bc1-afad-ee12982d211e" (UID: "b0875d60-67cb-4bc1-afad-ee12982d211e"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:03:28.622843 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.622824    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0875d60-67cb-4bc1-afad-ee12982d211e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0875d60-67cb-4bc1-afad-ee12982d211e" (UID: "b0875d60-67cb-4bc1-afad-ee12982d211e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:03:28.623000 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.622981    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0875d60-67cb-4bc1-afad-ee12982d211e-kube-api-access-596hm" (OuterVolumeSpecName: "kube-api-access-596hm") pod "b0875d60-67cb-4bc1-afad-ee12982d211e" (UID: "b0875d60-67cb-4bc1-afad-ee12982d211e"). InnerVolumeSpecName "kube-api-access-596hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:03:28.721609 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.721585    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596hm\" (UniqueName: \"kubernetes.io/projected/b0875d60-67cb-4bc1-afad-ee12982d211e-kube-api-access-596hm\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:03:28.721712 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.721609    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0875d60-67cb-4bc1-afad-ee12982d211e-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:03:28.721712 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.721624    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0875d60-67cb-4bc1-afad-ee12982d211e-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:03:28.721712 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:28.721638    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0875d60-67cb-4bc1-afad-ee12982d211e-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:03:29.211933 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.211895    2543 generic.go:358] "Generic (PLEG): container finished" podID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerID="39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450" exitCode=0
Apr 24 23:03:29.212087 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.211989    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"
Apr 24 23:03:29.212087 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.211998    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" event={"ID":"b0875d60-67cb-4bc1-afad-ee12982d211e","Type":"ContainerDied","Data":"39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450"}
Apr 24 23:03:29.212087 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.212073    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw" event={"ID":"b0875d60-67cb-4bc1-afad-ee12982d211e","Type":"ContainerDied","Data":"0cbaea4c7682bf83788da6373887e06987c54e068cef266e6087e775e4883afd"}
Apr 24 23:03:29.212196 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.212099    2543 scope.go:117] "RemoveContainer" containerID="1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c"
Apr 24 23:03:29.220189 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.220173    2543 scope.go:117] "RemoveContainer" containerID="39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450"
Apr 24 23:03:29.228578 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.228557    2543 scope.go:117] "RemoveContainer" containerID="133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e"
Apr 24 23:03:29.229673 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.229650    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"]
Apr 24 23:03:29.232850 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.232827    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-c5kzw"]
Apr 24 23:03:29.236436 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.236421    2543 scope.go:117] "RemoveContainer" containerID="1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c"
Apr 24 23:03:29.236681 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:03:29.236663    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c\": container with ID starting with 1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c not found: ID does not exist" containerID="1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c"
Apr 24 23:03:29.236741 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.236690    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c"} err="failed to get container status \"1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c\": rpc error: code = NotFound desc = could not find container \"1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c\": container with ID starting with 1f822ac83ef62cad5ef011ac83b4528c6895b1bf57351a27347d9239f1c9eb1c not found: ID does not exist"
Apr 24 23:03:29.236741 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.236706    2543 scope.go:117] "RemoveContainer" containerID="39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450"
Apr 24 23:03:29.236918 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:03:29.236904    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450\": container with ID starting with 39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450 not found: ID does not exist" containerID="39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450"
Apr 24 23:03:29.236963 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.236922    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450"} err="failed to get container status \"39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450\": rpc error: code = NotFound desc = could not find container \"39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450\": container with ID starting with 39b3e775d0bb495080f7fef3adaf6df3b1fdbb98692e4f1a0b72a98ad185c450 not found: ID does not exist"
Apr 24 23:03:29.236963 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.236936    2543 scope.go:117] "RemoveContainer" containerID="133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e"
Apr 24 23:03:29.237112 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:03:29.237091    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e\": container with ID starting with 133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e not found: ID does not exist" containerID="133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e"
Apr 24 23:03:29.237162 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:29.237121    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e"} err="failed to get container status \"133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e\": rpc error: code = NotFound desc = could not find container \"133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e\": container with ID starting with 133e732027567079f5cc417103d851866037980f66082334cc00ef36879c752e not found: ID does not exist"
Apr 24 23:03:30.216835 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:30.216804    2543 generic.go:358] "Generic (PLEG): container finished" podID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerID="e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b" exitCode=0
Apr 24 23:03:30.217252 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:30.216882    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" event={"ID":"11cd5412-257f-43ce-adad-6006cb9e2c12","Type":"ContainerDied","Data":"e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b"}
Apr 24 23:03:31.055494 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:31.055460    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" path="/var/lib/kubelet/pods/b0875d60-67cb-4bc1-afad-ee12982d211e/volumes"
Apr 24 23:03:31.224820 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:31.224784    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" event={"ID":"11cd5412-257f-43ce-adad-6006cb9e2c12","Type":"ContainerStarted","Data":"70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4"}
Apr 24 23:03:31.225202 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:31.224829    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" event={"ID":"11cd5412-257f-43ce-adad-6006cb9e2c12","Type":"ContainerStarted","Data":"162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7"}
Apr 24 23:03:31.225202 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:31.225114    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:31.245958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:31.245912    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podStartSLOduration=7.245899023 podStartE2EDuration="7.245899023s" podCreationTimestamp="2026-04-24 23:03:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:03:31.244648563 +0000 UTC m=+2014.671046093" watchObservedRunningTime="2026-04-24 23:03:31.245899023 +0000 UTC m=+2014.672296553"
Apr 24 23:03:32.229392 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:32.229361    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:32.230204 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:32.230180    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Apr 24 23:03:33.231663 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:33.231626    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Apr 24 23:03:38.236915 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:38.236889    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:03:38.237378 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:38.237303    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Apr 24 23:03:48.238305 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:48.238267    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Apr 24 23:03:58.237771 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:03:58.237729    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Apr 24 23:04:08.238145 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:08.238049    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Apr 24 23:04:18.237784 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:18.237744    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Apr 24 23:04:28.237873 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:28.237831    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Apr 24 23:04:38.238453 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:38.238420    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:04:44.523858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.523823    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"]
Apr 24 23:04:44.524333 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.524288    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" containerID="cri-o://162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7" gracePeriod=30
Apr 24 23:04:44.524500 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.524456    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kube-rbac-proxy" containerID="cri-o://70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4" gracePeriod=30
Apr 24 23:04:44.597823 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.597800    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"]
Apr 24 23:04:44.598170 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.598153    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kube-rbac-proxy"
Apr 24 23:04:44.598253 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.598173    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kube-rbac-proxy"
Apr 24 23:04:44.598253 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.598186    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container"
Apr 24 23:04:44.598253 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.598194    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container"
Apr 24 23:04:44.598253 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.598217    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="storage-initializer"
Apr 24 23:04:44.598253 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.598226    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="storage-initializer"
Apr 24 23:04:44.598537 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.598332    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kserve-container"
Apr 24 23:04:44.598537 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.598368    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0875d60-67cb-4bc1-afad-ee12982d211e" containerName="kube-rbac-proxy"
Apr 24 23:04:44.601517 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.601499    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.603898 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.603883    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Apr 24 23:04:44.603983 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.603892    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Apr 24 23:04:44.612486 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.612466    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"]
Apr 24 23:04:44.766762 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.766731    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db7b114-50f0-4824-adda-8f060a3bd4a3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.766909 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.766779    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db7b114-50f0-4824-adda-8f060a3bd4a3-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.766909 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.766797    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtk8\" (UniqueName: \"kubernetes.io/projected/2db7b114-50f0-4824-adda-8f060a3bd4a3-kube-api-access-ndtk8\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.766909 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.766882    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db7b114-50f0-4824-adda-8f060a3bd4a3-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.868033 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.867958    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db7b114-50f0-4824-adda-8f060a3bd4a3-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.868033 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.867990    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtk8\" (UniqueName: \"kubernetes.io/projected/2db7b114-50f0-4824-adda-8f060a3bd4a3-kube-api-access-ndtk8\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.868033 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.868021    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db7b114-50f0-4824-adda-8f060a3bd4a3-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.868280 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.868094    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db7b114-50f0-4824-adda-8f060a3bd4a3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.868280 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:04:44.868202    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Apr 24 23:04:44.868280 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:04:44.868255    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db7b114-50f0-4824-adda-8f060a3bd4a3-proxy-tls podName:2db7b114-50f0-4824-adda-8f060a3bd4a3 nodeName:}" failed. No retries permitted until 2026-04-24 23:04:45.368237137 +0000 UTC m=+2088.794634644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2db7b114-50f0-4824-adda-8f060a3bd4a3-proxy-tls") pod "isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" (UID: "2db7b114-50f0-4824-adda-8f060a3bd4a3") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Apr 24 23:04:44.868463 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.868443    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db7b114-50f0-4824-adda-8f060a3bd4a3-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.868598 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.868580    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db7b114-50f0-4824-adda-8f060a3bd4a3-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:44.876736 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:44.876717    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtk8\" (UniqueName: \"kubernetes.io/projected/2db7b114-50f0-4824-adda-8f060a3bd4a3-kube-api-access-ndtk8\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:45.373034 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:45.372996    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db7b114-50f0-4824-adda-8f060a3bd4a3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:45.375536 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:45.375516    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db7b114-50f0-4824-adda-8f060a3bd4a3-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:45.433473 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:45.433442    2543 generic.go:358] "Generic (PLEG): container finished" podID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerID="70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4" exitCode=2
Apr 24 23:04:45.433620 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:45.433496    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" event={"ID":"11cd5412-257f-43ce-adad-6006cb9e2c12","Type":"ContainerDied","Data":"70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4"}
Apr 24 23:04:45.511088 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:45.511042    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:45.632733 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:45.632590    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"]
Apr 24 23:04:45.635426 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:04:45.635391    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db7b114_50f0_4824_adda_8f060a3bd4a3.slice/crio-a1f33e7a2975282aa1c56b9c41edf359bcbf7fdf8e51586d05075a7d1ce9652d WatchSource:0}: Error finding container a1f33e7a2975282aa1c56b9c41edf359bcbf7fdf8e51586d05075a7d1ce9652d: Status 404 returned error can't find the container with id a1f33e7a2975282aa1c56b9c41edf359bcbf7fdf8e51586d05075a7d1ce9652d
Apr 24 23:04:45.637480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:45.637465    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 23:04:46.438240 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:46.438197    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" event={"ID":"2db7b114-50f0-4824-adda-8f060a3bd4a3","Type":"ContainerStarted","Data":"53aa47192437ad2db384543e5db705e4893c63ae416da5dac892939a55a3b209"}
Apr 24 23:04:46.438240 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:46.438238    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" event={"ID":"2db7b114-50f0-4824-adda-8f060a3bd4a3","Type":"ContainerStarted","Data":"a1f33e7a2975282aa1c56b9c41edf359bcbf7fdf8e51586d05075a7d1ce9652d"}
Apr 24 23:04:48.232403 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.232359    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Apr 24 23:04:48.237677 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.237653    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Apr 24 23:04:48.856159 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.856129    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:04:48.894893 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.894873    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11cd5412-257f-43ce-adad-6006cb9e2c12-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"11cd5412-257f-43ce-adad-6006cb9e2c12\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") "
Apr 24 23:04:48.895009 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.894914    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqwmg\" (UniqueName: \"kubernetes.io/projected/11cd5412-257f-43ce-adad-6006cb9e2c12-kube-api-access-jqwmg\") pod \"11cd5412-257f-43ce-adad-6006cb9e2c12\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") "
Apr 24 23:04:48.895009 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.894937    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11cd5412-257f-43ce-adad-6006cb9e2c12-proxy-tls\") pod \"11cd5412-257f-43ce-adad-6006cb9e2c12\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") "
Apr 24 23:04:48.895009 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.894972    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11cd5412-257f-43ce-adad-6006cb9e2c12-kserve-provision-location\") pod \"11cd5412-257f-43ce-adad-6006cb9e2c12\" (UID: \"11cd5412-257f-43ce-adad-6006cb9e2c12\") "
Apr 24 23:04:48.895235 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.895208    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cd5412-257f-43ce-adad-6006cb9e2c12-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "11cd5412-257f-43ce-adad-6006cb9e2c12" (UID: "11cd5412-257f-43ce-adad-6006cb9e2c12"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:04:48.895416 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.895392    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11cd5412-257f-43ce-adad-6006cb9e2c12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11cd5412-257f-43ce-adad-6006cb9e2c12" (UID: "11cd5412-257f-43ce-adad-6006cb9e2c12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:04:48.897068 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.897046    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cd5412-257f-43ce-adad-6006cb9e2c12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11cd5412-257f-43ce-adad-6006cb9e2c12" (UID: "11cd5412-257f-43ce-adad-6006cb9e2c12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:04:48.897170 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.897149    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cd5412-257f-43ce-adad-6006cb9e2c12-kube-api-access-jqwmg" (OuterVolumeSpecName: "kube-api-access-jqwmg") pod "11cd5412-257f-43ce-adad-6006cb9e2c12" (UID: "11cd5412-257f-43ce-adad-6006cb9e2c12"). InnerVolumeSpecName "kube-api-access-jqwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:04:48.995740 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.995686    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11cd5412-257f-43ce-adad-6006cb9e2c12-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:04:48.995740 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.995708    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqwmg\" (UniqueName: \"kubernetes.io/projected/11cd5412-257f-43ce-adad-6006cb9e2c12-kube-api-access-jqwmg\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:04:48.995740 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.995718    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11cd5412-257f-43ce-adad-6006cb9e2c12-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:04:48.995740 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:48.995726    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11cd5412-257f-43ce-adad-6006cb9e2c12-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:04:49.448909 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.448839    2543 generic.go:358] "Generic (PLEG): container finished" podID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerID="162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7" exitCode=0
Apr 24 23:04:49.448909 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.448888    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" event={"ID":"11cd5412-257f-43ce-adad-6006cb9e2c12","Type":"ContainerDied","Data":"162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7"}
Apr 24 23:04:49.449357 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.448927    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7" event={"ID":"11cd5412-257f-43ce-adad-6006cb9e2c12","Type":"ContainerDied","Data":"3c65d7998f0587b18e115cce2381002dc58698f863540cb5a3ead61ca8b36927"}
Apr 24 23:04:49.449357 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.448927    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"
Apr 24 23:04:49.449357 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.448941    2543 scope.go:117] "RemoveContainer" containerID="70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4"
Apr 24 23:04:49.450328 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.450306    2543 generic.go:358] "Generic (PLEG): container finished" podID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerID="53aa47192437ad2db384543e5db705e4893c63ae416da5dac892939a55a3b209" exitCode=0
Apr 24 23:04:49.450450 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.450358    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" event={"ID":"2db7b114-50f0-4824-adda-8f060a3bd4a3","Type":"ContainerDied","Data":"53aa47192437ad2db384543e5db705e4893c63ae416da5dac892939a55a3b209"}
Apr 24 23:04:49.456532 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.456514    2543 scope.go:117] "RemoveContainer" containerID="162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7"
Apr 24 23:04:49.463479 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.463451    2543 scope.go:117] "RemoveContainer" containerID="e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b"
Apr 24 23:04:49.470264 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.470236    2543 scope.go:117] "RemoveContainer" containerID="70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4"
Apr 24 23:04:49.470548 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:04:49.470524    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4\": container with ID starting with 70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4 not found: ID does not exist" containerID="70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4"
Apr 24 23:04:49.470650 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.470559    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4"} err="failed to get container status \"70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4\": rpc error: code = NotFound desc = could not find container \"70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4\": container with ID starting with 70c390a7b4a4bde3bfe4fd72620386568fa190d8d2a9bacf2173a252565d60c4 not found: ID does not exist"
Apr 24 23:04:49.470650 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.470577    2543 scope.go:117] "RemoveContainer" containerID="162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7"
Apr 24 23:04:49.470825 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:04:49.470806    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7\": container with ID starting with 162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7 not found: ID does not exist" containerID="162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7"
Apr 24 23:04:49.470888 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.470830    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7"} err="failed to get container status \"162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7\": rpc error: code = NotFound desc = could not find container \"162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7\": container with ID starting with 162ae66e5a0666e7f8d9bb7b3bd8d00012c2c04e20c860d6f58074e612d86cb7 not found: ID does not exist"
Apr 24 23:04:49.470888 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.470847    2543 scope.go:117] "RemoveContainer" containerID="e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b"
Apr 24 23:04:49.471101 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:04:49.471083    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b\": container with ID starting with e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b not found: ID does not exist" containerID="e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b"
Apr 24 23:04:49.471139 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.471109    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b"} err="failed to get container status \"e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b\": rpc error: code = NotFound desc = could not find container \"e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b\": container with ID starting with e673f9578cd42a237b51195f88cb3fcc28e21bbd77a90c4fc8e00a80c1626e1b not found: ID does not exist"
Apr 24 23:04:49.496516 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.496495    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"]
Apr 24 23:04:49.502919 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:49.502899    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-5rpj7"]
Apr 24 23:04:50.455804 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:50.455769    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" event={"ID":"2db7b114-50f0-4824-adda-8f060a3bd4a3","Type":"ContainerStarted","Data":"75df498fe4cf8950e19b855f0ff4b05f976eaf33db7474776e52f6e169872ad7"}
Apr 24 23:04:50.455804 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:50.455806    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" event={"ID":"2db7b114-50f0-4824-adda-8f060a3bd4a3","Type":"ContainerStarted","Data":"5a0050fcf3a2404028f63ef97fe0ee55a5056e026211ca98d85afe9dde4e7873"}
Apr 24 23:04:50.456316 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:50.456068    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:50.477806 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:50.477760    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podStartSLOduration=6.477748053 podStartE2EDuration="6.477748053s" podCreationTimestamp="2026-04-24 23:04:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:04:50.476395423 +0000 UTC m=+2093.902792952" watchObservedRunningTime="2026-04-24 23:04:50.477748053 +0000 UTC m=+2093.904145582"
Apr 24 23:04:51.055196 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:51.055164    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" path="/var/lib/kubelet/pods/11cd5412-257f-43ce-adad-6006cb9e2c12/volumes"
Apr 24 23:04:51.459755 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:51.459723    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:51.460927 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:51.460899    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:04:52.462772 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:52.462740    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:04:57.466633 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:57.466602    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:04:57.467202 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:57.467174    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:04:57.509726 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:57.509700    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:04:57.513823 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:04:57.513804    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:05:07.467608 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:05:07.467571    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:05:17.467292 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:05:17.467256    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:05:27.467592 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:05:27.467555    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:05:37.467758 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:05:37.467681    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:05:47.467723 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:05:47.467680    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:05:57.467556 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:05:57.467509    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:06:07.468185 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:07.468149    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:06:14.729462 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.729428    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"]
Apr 24 23:06:14.729846 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.729747    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" containerID="cri-o://5a0050fcf3a2404028f63ef97fe0ee55a5056e026211ca98d85afe9dde4e7873" gracePeriod=30
Apr 24 23:06:14.729846 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.729793    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kube-rbac-proxy" containerID="cri-o://75df498fe4cf8950e19b855f0ff4b05f976eaf33db7474776e52f6e169872ad7" gracePeriod=30
Apr 24 23:06:14.873246 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.873214    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"]
Apr 24 23:06:14.873536 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.873524    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kube-rbac-proxy"
Apr 24 23:06:14.873591 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.873538    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kube-rbac-proxy"
Apr 24 23:06:14.873591 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.873551    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="storage-initializer"
Apr 24 23:06:14.873591 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.873557    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="storage-initializer"
Apr 24 23:06:14.873591 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.873575    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container"
Apr 24 23:06:14.873591 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.873580    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container"
Apr 24 23:06:14.873747 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.873623    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kserve-container"
Apr 24 23:06:14.873747 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.873630    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="11cd5412-257f-43ce-adad-6006cb9e2c12" containerName="kube-rbac-proxy"
Apr 24 23:06:14.876458 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.876443    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:14.878759 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.878723    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Apr 24 23:06:14.878934 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.878914    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Apr 24 23:06:14.886581 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:14.886560    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"]
Apr 24 23:06:15.011721 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.011642    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51b2381-c422-413a-9525-2de250f08f4b-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.011721 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.011703    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b2381-c422-413a-9525-2de250f08f4b-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.011894 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.011776    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51b2381-c422-413a-9525-2de250f08f4b-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.011894 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.011814    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtzwj\" (UniqueName: \"kubernetes.io/projected/c51b2381-c422-413a-9525-2de250f08f4b-kube-api-access-wtzwj\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.112661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.112633    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b2381-c422-413a-9525-2de250f08f4b-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.112808 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.112675    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51b2381-c422-413a-9525-2de250f08f4b-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.112808 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.112693    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wtzwj\" (UniqueName: \"kubernetes.io/projected/c51b2381-c422-413a-9525-2de250f08f4b-kube-api-access-wtzwj\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.112808 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.112728    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51b2381-c422-413a-9525-2de250f08f4b-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.113077 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.113053    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b2381-c422-413a-9525-2de250f08f4b-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.113300 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.113284    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51b2381-c422-413a-9525-2de250f08f4b-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.115176 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.115159    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51b2381-c422-413a-9525-2de250f08f4b-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.120761 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.120738    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtzwj\" (UniqueName: \"kubernetes.io/projected/c51b2381-c422-413a-9525-2de250f08f4b-kube-api-access-wtzwj\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.187828 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.187804    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:15.305842 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.305818    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"]
Apr 24 23:06:15.308302 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:06:15.308276    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51b2381_c422_413a_9525_2de250f08f4b.slice/crio-d65429628f7625f5439c9dcc6ee835b09a7ce923a4b9fbd9464d46681664cc00 WatchSource:0}: Error finding container d65429628f7625f5439c9dcc6ee835b09a7ce923a4b9fbd9464d46681664cc00: Status 404 returned error can't find the container with id d65429628f7625f5439c9dcc6ee835b09a7ce923a4b9fbd9464d46681664cc00
Apr 24 23:06:15.696853 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.696758    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" event={"ID":"c51b2381-c422-413a-9525-2de250f08f4b","Type":"ContainerStarted","Data":"41e5850f1f7324bf6c8926cac538c47c35c5c0c8773da6c4b4afb923e615cc2e"}
Apr 24 23:06:15.696853 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.696801    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" event={"ID":"c51b2381-c422-413a-9525-2de250f08f4b","Type":"ContainerStarted","Data":"d65429628f7625f5439c9dcc6ee835b09a7ce923a4b9fbd9464d46681664cc00"}
Apr 24 23:06:15.698613 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.698589    2543 generic.go:358] "Generic (PLEG): container finished" podID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerID="75df498fe4cf8950e19b855f0ff4b05f976eaf33db7474776e52f6e169872ad7" exitCode=2
Apr 24 23:06:15.698728 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:15.698626    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" event={"ID":"2db7b114-50f0-4824-adda-8f060a3bd4a3","Type":"ContainerDied","Data":"75df498fe4cf8950e19b855f0ff4b05f976eaf33db7474776e52f6e169872ad7"}
Apr 24 23:06:17.462944 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:17.462910    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Apr 24 23:06:17.467209 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:17.467184    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Apr 24 23:06:19.711560 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:19.711522    2543 generic.go:358] "Generic (PLEG): container finished" podID="c51b2381-c422-413a-9525-2de250f08f4b" containerID="41e5850f1f7324bf6c8926cac538c47c35c5c0c8773da6c4b4afb923e615cc2e" exitCode=0
Apr 24 23:06:19.711918 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:19.711598    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" event={"ID":"c51b2381-c422-413a-9525-2de250f08f4b","Type":"ContainerDied","Data":"41e5850f1f7324bf6c8926cac538c47c35c5c0c8773da6c4b4afb923e615cc2e"}
Apr 24 23:06:19.713480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:19.713460    2543 generic.go:358] "Generic (PLEG): container finished" podID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerID="5a0050fcf3a2404028f63ef97fe0ee55a5056e026211ca98d85afe9dde4e7873" exitCode=0
Apr 24 23:06:19.713575 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:19.713503    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" event={"ID":"2db7b114-50f0-4824-adda-8f060a3bd4a3","Type":"ContainerDied","Data":"5a0050fcf3a2404028f63ef97fe0ee55a5056e026211ca98d85afe9dde4e7873"}
Apr 24 23:06:19.873968 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:19.873946    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:06:20.051450 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.051332    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db7b114-50f0-4824-adda-8f060a3bd4a3-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"2db7b114-50f0-4824-adda-8f060a3bd4a3\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") "
Apr 24 23:06:20.051621 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.051456    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndtk8\" (UniqueName: \"kubernetes.io/projected/2db7b114-50f0-4824-adda-8f060a3bd4a3-kube-api-access-ndtk8\") pod \"2db7b114-50f0-4824-adda-8f060a3bd4a3\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") "
Apr 24 23:06:20.051621 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.051502    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db7b114-50f0-4824-adda-8f060a3bd4a3-kserve-provision-location\") pod \"2db7b114-50f0-4824-adda-8f060a3bd4a3\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") "
Apr 24 23:06:20.051621 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.051551    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db7b114-50f0-4824-adda-8f060a3bd4a3-proxy-tls\") pod \"2db7b114-50f0-4824-adda-8f060a3bd4a3\" (UID: \"2db7b114-50f0-4824-adda-8f060a3bd4a3\") "
Apr 24 23:06:20.051787 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.051651    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db7b114-50f0-4824-adda-8f060a3bd4a3-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "2db7b114-50f0-4824-adda-8f060a3bd4a3" (UID: "2db7b114-50f0-4824-adda-8f060a3bd4a3"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:06:20.051859 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.051845    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db7b114-50f0-4824-adda-8f060a3bd4a3-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:06:20.051859 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.051846    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db7b114-50f0-4824-adda-8f060a3bd4a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2db7b114-50f0-4824-adda-8f060a3bd4a3" (UID: "2db7b114-50f0-4824-adda-8f060a3bd4a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:06:20.053678 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.053645    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db7b114-50f0-4824-adda-8f060a3bd4a3-kube-api-access-ndtk8" (OuterVolumeSpecName: "kube-api-access-ndtk8") pod "2db7b114-50f0-4824-adda-8f060a3bd4a3" (UID: "2db7b114-50f0-4824-adda-8f060a3bd4a3"). InnerVolumeSpecName "kube-api-access-ndtk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:06:20.053678 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.053661    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db7b114-50f0-4824-adda-8f060a3bd4a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2db7b114-50f0-4824-adda-8f060a3bd4a3" (UID: "2db7b114-50f0-4824-adda-8f060a3bd4a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:06:20.153544 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.153237    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db7b114-50f0-4824-adda-8f060a3bd4a3-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:06:20.153544 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.153265    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtk8\" (UniqueName: \"kubernetes.io/projected/2db7b114-50f0-4824-adda-8f060a3bd4a3-kube-api-access-ndtk8\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:06:20.153544 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.153282    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2db7b114-50f0-4824-adda-8f060a3bd4a3-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:06:20.718269 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.718230    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" event={"ID":"c51b2381-c422-413a-9525-2de250f08f4b","Type":"ContainerStarted","Data":"456eef9a19661f2ab0f659be798a1be1d6c30421ccb5ff2cddadaa1102af5b60"}
Apr 24 23:06:20.718696 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.718280    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" event={"ID":"c51b2381-c422-413a-9525-2de250f08f4b","Type":"ContainerStarted","Data":"547c793beec65bd76f4154fff777cf6c604eeae3ad3aede6ed9f59f1f5d9a3f6"}
Apr 24 23:06:20.718696 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.718538    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:20.719892 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.719869    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw" event={"ID":"2db7b114-50f0-4824-adda-8f060a3bd4a3","Type":"ContainerDied","Data":"a1f33e7a2975282aa1c56b9c41edf359bcbf7fdf8e51586d05075a7d1ce9652d"}
Apr 24 23:06:20.720011 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.719905    2543 scope.go:117] "RemoveContainer" containerID="75df498fe4cf8950e19b855f0ff4b05f976eaf33db7474776e52f6e169872ad7"
Apr 24 23:06:20.720011 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.719920    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"
Apr 24 23:06:20.729009 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.728994    2543 scope.go:117] "RemoveContainer" containerID="5a0050fcf3a2404028f63ef97fe0ee55a5056e026211ca98d85afe9dde4e7873"
Apr 24 23:06:20.736570 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.736537    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" podStartSLOduration=6.736525811 podStartE2EDuration="6.736525811s" podCreationTimestamp="2026-04-24 23:06:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:06:20.735731423 +0000 UTC m=+2184.162128953" watchObservedRunningTime="2026-04-24 23:06:20.736525811 +0000 UTC m=+2184.162923320"
Apr 24 23:06:20.736677 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.736569    2543 scope.go:117] "RemoveContainer" containerID="53aa47192437ad2db384543e5db705e4893c63ae416da5dac892939a55a3b209"
Apr 24 23:06:20.747867 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.747844    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"]
Apr 24 23:06:20.751539 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:20.751521    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-fzvxw"]
Apr 24 23:06:21.055663 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:21.055598    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" path="/var/lib/kubelet/pods/2db7b114-50f0-4824-adda-8f060a3bd4a3/volumes"
Apr 24 23:06:21.724267 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:21.724238    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:27.733024 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:27.732995    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:06:57.733969 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:06:57.733930    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 24 23:07:07.734197 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:07.734114    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 24 23:07:17.733796 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:17.733756    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 24 23:07:27.733989 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:27.733943    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 24 23:07:35.055774 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:35.055743    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:07:44.947097 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:44.947057    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"]
Apr 24 23:07:44.947619 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:44.947488    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kserve-container" containerID="cri-o://547c793beec65bd76f4154fff777cf6c604eeae3ad3aede6ed9f59f1f5d9a3f6" gracePeriod=30
Apr 24 23:07:44.947619 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:44.947523    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kube-rbac-proxy" containerID="cri-o://456eef9a19661f2ab0f659be798a1be1d6c30421ccb5ff2cddadaa1102af5b60" gracePeriod=30
Apr 24 23:07:45.052252 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.052207    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Apr 24 23:07:45.064427 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.064400    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"]
Apr 24 23:07:45.064682 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.064671    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container"
Apr 24 23:07:45.064724 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.064684    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container"
Apr 24 23:07:45.064724 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.064693    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kube-rbac-proxy"
Apr 24 23:07:45.064724 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.064698    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kube-rbac-proxy"
Apr 24 23:07:45.064724 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.064712    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="storage-initializer"
Apr 24 23:07:45.064724 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.064717    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="storage-initializer"
Apr 24 23:07:45.064877 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.064770    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kserve-container"
Apr 24 23:07:45.064877 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.064779    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2db7b114-50f0-4824-adda-8f060a3bd4a3" containerName="kube-rbac-proxy"
Apr 24 23:07:45.067588 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.067573    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.069917 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.069895    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Apr 24 23:07:45.070036 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.069935    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Apr 24 23:07:45.075326 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.075301    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"]
Apr 24 23:07:45.145749 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.145723    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0355fe3d-1446-46b6-9804-132c10720573-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.145848 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.145813    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0355fe3d-1446-46b6-9804-132c10720573-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.145978 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.145952    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g67kk\" (UniqueName: \"kubernetes.io/projected/0355fe3d-1446-46b6-9804-132c10720573-kube-api-access-g67kk\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.146120 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.146104    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0355fe3d-1446-46b6-9804-132c10720573-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.246926 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.246902    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0355fe3d-1446-46b6-9804-132c10720573-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.247064 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.246934    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g67kk\" (UniqueName: \"kubernetes.io/projected/0355fe3d-1446-46b6-9804-132c10720573-kube-api-access-g67kk\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.247064 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.246982    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0355fe3d-1446-46b6-9804-132c10720573-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.247064 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.247003    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0355fe3d-1446-46b6-9804-132c10720573-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.247431 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.247403    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0355fe3d-1446-46b6-9804-132c10720573-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.247766 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.247743    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0355fe3d-1446-46b6-9804-132c10720573-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.249287 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.249268    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0355fe3d-1446-46b6-9804-132c10720573-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.257635 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.257612    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g67kk\" (UniqueName: \"kubernetes.io/projected/0355fe3d-1446-46b6-9804-132c10720573-kube-api-access-g67kk\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.378820 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.378799    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:45.499479 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.499455    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"]
Apr 24 23:07:45.501840 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:07:45.501810    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0355fe3d_1446_46b6_9804_132c10720573.slice/crio-a24fd4d91c3e7394b7471969a01f07840499966ac4c634d589a7b615452997b9 WatchSource:0}: Error finding container a24fd4d91c3e7394b7471969a01f07840499966ac4c634d589a7b615452997b9: Status 404 returned error can't find the container with id a24fd4d91c3e7394b7471969a01f07840499966ac4c634d589a7b615452997b9
Apr 24 23:07:45.967331 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.967299    2543 generic.go:358] "Generic (PLEG): container finished" podID="c51b2381-c422-413a-9525-2de250f08f4b" containerID="456eef9a19661f2ab0f659be798a1be1d6c30421ccb5ff2cddadaa1102af5b60" exitCode=2
Apr 24 23:07:45.967751 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.967368    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" event={"ID":"c51b2381-c422-413a-9525-2de250f08f4b","Type":"ContainerDied","Data":"456eef9a19661f2ab0f659be798a1be1d6c30421ccb5ff2cddadaa1102af5b60"}
Apr 24 23:07:45.968811 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.968781    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" event={"ID":"0355fe3d-1446-46b6-9804-132c10720573","Type":"ContainerStarted","Data":"84d378b4ebccef9c24f0ba11b39d5d15ba9f153e8b18cacf3e8112cc31dbdb65"}
Apr 24 23:07:45.968811 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:45.968819    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" event={"ID":"0355fe3d-1446-46b6-9804-132c10720573","Type":"ContainerStarted","Data":"a24fd4d91c3e7394b7471969a01f07840499966ac4c634d589a7b615452997b9"}
Apr 24 23:07:47.727854 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:47.727814    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Apr 24 23:07:48.978803 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:48.978768    2543 generic.go:358] "Generic (PLEG): container finished" podID="c51b2381-c422-413a-9525-2de250f08f4b" containerID="547c793beec65bd76f4154fff777cf6c604eeae3ad3aede6ed9f59f1f5d9a3f6" exitCode=0
Apr 24 23:07:48.979119 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:48.978834    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" event={"ID":"c51b2381-c422-413a-9525-2de250f08f4b","Type":"ContainerDied","Data":"547c793beec65bd76f4154fff777cf6c604eeae3ad3aede6ed9f59f1f5d9a3f6"}
Apr 24 23:07:49.090886 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.090862    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:07:49.176925 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.176866    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51b2381-c422-413a-9525-2de250f08f4b-proxy-tls\") pod \"c51b2381-c422-413a-9525-2de250f08f4b\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") "
Apr 24 23:07:49.176925 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.176904    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtzwj\" (UniqueName: \"kubernetes.io/projected/c51b2381-c422-413a-9525-2de250f08f4b-kube-api-access-wtzwj\") pod \"c51b2381-c422-413a-9525-2de250f08f4b\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") "
Apr 24 23:07:49.177100 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.176945    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b2381-c422-413a-9525-2de250f08f4b-kserve-provision-location\") pod \"c51b2381-c422-413a-9525-2de250f08f4b\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") "
Apr 24 23:07:49.177100 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.176971    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51b2381-c422-413a-9525-2de250f08f4b-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"c51b2381-c422-413a-9525-2de250f08f4b\" (UID: \"c51b2381-c422-413a-9525-2de250f08f4b\") "
Apr 24 23:07:49.177314 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.177275    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51b2381-c422-413a-9525-2de250f08f4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c51b2381-c422-413a-9525-2de250f08f4b" (UID: "c51b2381-c422-413a-9525-2de250f08f4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:07:49.177420 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.177361    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51b2381-c422-413a-9525-2de250f08f4b-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "c51b2381-c422-413a-9525-2de250f08f4b" (UID: "c51b2381-c422-413a-9525-2de250f08f4b"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:07:49.178873 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.178851    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51b2381-c422-413a-9525-2de250f08f4b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c51b2381-c422-413a-9525-2de250f08f4b" (UID: "c51b2381-c422-413a-9525-2de250f08f4b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:07:49.178957 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.178930    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51b2381-c422-413a-9525-2de250f08f4b-kube-api-access-wtzwj" (OuterVolumeSpecName: "kube-api-access-wtzwj") pod "c51b2381-c422-413a-9525-2de250f08f4b" (UID: "c51b2381-c422-413a-9525-2de250f08f4b"). InnerVolumeSpecName "kube-api-access-wtzwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:07:49.278195 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.278166    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b2381-c422-413a-9525-2de250f08f4b-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:07:49.278329 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.278199    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c51b2381-c422-413a-9525-2de250f08f4b-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:07:49.278329 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.278214    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c51b2381-c422-413a-9525-2de250f08f4b-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:07:49.278329 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.278228    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtzwj\" (UniqueName: \"kubernetes.io/projected/c51b2381-c422-413a-9525-2de250f08f4b-kube-api-access-wtzwj\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:07:49.983164 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.983130    2543 generic.go:358] "Generic (PLEG): container finished" podID="0355fe3d-1446-46b6-9804-132c10720573" containerID="84d378b4ebccef9c24f0ba11b39d5d15ba9f153e8b18cacf3e8112cc31dbdb65" exitCode=0
Apr 24 23:07:49.983612 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.983202    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" event={"ID":"0355fe3d-1446-46b6-9804-132c10720573","Type":"ContainerDied","Data":"84d378b4ebccef9c24f0ba11b39d5d15ba9f153e8b18cacf3e8112cc31dbdb65"}
Apr 24 23:07:49.984879 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.984861    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb" event={"ID":"c51b2381-c422-413a-9525-2de250f08f4b","Type":"ContainerDied","Data":"d65429628f7625f5439c9dcc6ee835b09a7ce923a4b9fbd9464d46681664cc00"}
Apr 24 23:07:49.984962 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.984892    2543 scope.go:117] "RemoveContainer" containerID="456eef9a19661f2ab0f659be798a1be1d6c30421ccb5ff2cddadaa1102af5b60"
Apr 24 23:07:49.984962 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.984929    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"
Apr 24 23:07:49.993259 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:49.993233    2543 scope.go:117] "RemoveContainer" containerID="547c793beec65bd76f4154fff777cf6c604eeae3ad3aede6ed9f59f1f5d9a3f6"
Apr 24 23:07:50.002830 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:50.002803    2543 scope.go:117] "RemoveContainer" containerID="41e5850f1f7324bf6c8926cac538c47c35c5c0c8773da6c4b4afb923e615cc2e"
Apr 24 23:07:50.013508 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:50.013441    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"]
Apr 24 23:07:50.016682 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:50.016662    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-hvnxb"]
Apr 24 23:07:50.989353 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:50.989313    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" event={"ID":"0355fe3d-1446-46b6-9804-132c10720573","Type":"ContainerStarted","Data":"457a43c860569c9dec32b39c29b01ac5cee374022bf78b65b3b7cd6b0e9a54ae"}
Apr 24 23:07:50.989775 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:50.989377    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" event={"ID":"0355fe3d-1446-46b6-9804-132c10720573","Type":"ContainerStarted","Data":"7bc703361c62f13cd949b955ea5e8763a4a2afd624adf233777d5720b5017bd4"}
Apr 24 23:07:50.989775 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:50.989597    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:51.006628 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:51.006588    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" podStartSLOduration=6.00657625 podStartE2EDuration="6.00657625s" podCreationTimestamp="2026-04-24 23:07:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:07:51.005492806 +0000 UTC m=+2274.431890337" watchObservedRunningTime="2026-04-24 23:07:51.00657625 +0000 UTC m=+2274.432973779"
Apr 24 23:07:51.054753 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:51.054719    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c51b2381-c422-413a-9525-2de250f08f4b" path="/var/lib/kubelet/pods/c51b2381-c422-413a-9525-2de250f08f4b/volumes"
Apr 24 23:07:51.993122 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:51.993089    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:07:58.001435 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:07:58.001405    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:08:28.002572 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:08:28.002532    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 24 23:08:38.002176 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:08:38.002081    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 24 23:08:48.002098 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:08:48.002056    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 24 23:08:58.002310 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:08:58.002262    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 24 23:09:08.005749 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:08.005720    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:09:15.150782 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.150746    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"]
Apr 24 23:09:15.151248 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.151056    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kserve-container" containerID="cri-o://7bc703361c62f13cd949b955ea5e8763a4a2afd624adf233777d5720b5017bd4" gracePeriod=30
Apr 24 23:09:15.151248 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.151084    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kube-rbac-proxy" containerID="cri-o://457a43c860569c9dec32b39c29b01ac5cee374022bf78b65b3b7cd6b0e9a54ae" gracePeriod=30
Apr 24 23:09:15.538352 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.538297    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"]
Apr 24 23:09:15.538656 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.538642    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="storage-initializer"
Apr 24 23:09:15.538703 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.538658    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="storage-initializer"
Apr 24 23:09:15.538703 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.538670    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kube-rbac-proxy"
Apr 24 23:09:15.538703 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.538676    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kube-rbac-proxy"
Apr 24 23:09:15.538703 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.538689    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kserve-container"
Apr 24 23:09:15.538703 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.538694    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kserve-container"
Apr 24 23:09:15.538857 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.538754    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kube-rbac-proxy"
Apr 24 23:09:15.538857 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.538764    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="c51b2381-c422-413a-9525-2de250f08f4b" containerName="kserve-container"
Apr 24 23:09:15.541796 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.541780    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.544113 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.544089    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Apr 24 23:09:15.544233 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.544147    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Apr 24 23:09:15.549698 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.549675    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"]
Apr 24 23:09:15.624897 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.624870    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7p4\" (UniqueName: \"kubernetes.io/projected/e5df282a-0b72-430f-a9a8-740d68358161-kube-api-access-5k7p4\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.625007 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.624900    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5df282a-0b72-430f-a9a8-740d68358161-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.625007 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.624934    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5df282a-0b72-430f-a9a8-740d68358161-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.625086 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.625040    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5df282a-0b72-430f-a9a8-740d68358161-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.725912 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.725885    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7p4\" (UniqueName: \"kubernetes.io/projected/e5df282a-0b72-430f-a9a8-740d68358161-kube-api-access-5k7p4\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.726012 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.725917    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5df282a-0b72-430f-a9a8-740d68358161-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.726012 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.725939    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5df282a-0b72-430f-a9a8-740d68358161-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.726012 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.725996    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5df282a-0b72-430f-a9a8-740d68358161-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.726371 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.726324    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5df282a-0b72-430f-a9a8-740d68358161-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.726639 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.726621    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5df282a-0b72-430f-a9a8-740d68358161-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.728486 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.728463    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5df282a-0b72-430f-a9a8-740d68358161-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.733670 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.733645    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7p4\" (UniqueName: \"kubernetes.io/projected/e5df282a-0b72-430f-a9a8-740d68358161-kube-api-access-5k7p4\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.852129 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.852060    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:15.970019 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:15.969989    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"]
Apr 24 23:09:15.972402 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:09:15.972376    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df282a_0b72_430f_a9a8_740d68358161.slice/crio-9983652f7bd2cec0d00d0fda3cdaaef7632337bd9a3c771a7353dc609e0106ae WatchSource:0}: Error finding container 9983652f7bd2cec0d00d0fda3cdaaef7632337bd9a3c771a7353dc609e0106ae: Status 404 returned error can't find the container with id 9983652f7bd2cec0d00d0fda3cdaaef7632337bd9a3c771a7353dc609e0106ae
Apr 24 23:09:16.233026 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:16.232990    2543 generic.go:358] "Generic (PLEG): container finished" podID="0355fe3d-1446-46b6-9804-132c10720573" containerID="457a43c860569c9dec32b39c29b01ac5cee374022bf78b65b3b7cd6b0e9a54ae" exitCode=2
Apr 24 23:09:16.233516 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:16.233061    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" event={"ID":"0355fe3d-1446-46b6-9804-132c10720573","Type":"ContainerDied","Data":"457a43c860569c9dec32b39c29b01ac5cee374022bf78b65b3b7cd6b0e9a54ae"}
Apr 24 23:09:16.234398 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:16.234374    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" event={"ID":"e5df282a-0b72-430f-a9a8-740d68358161","Type":"ContainerStarted","Data":"78e86888f2ff648b9d4f8e68211d41dde845674110baae500520342d3599091a"}
Apr 24 23:09:16.234519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:16.234402    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" event={"ID":"e5df282a-0b72-430f-a9a8-740d68358161","Type":"ContainerStarted","Data":"9983652f7bd2cec0d00d0fda3cdaaef7632337bd9a3c771a7353dc609e0106ae"}
Apr 24 23:09:17.996488 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:17.996439    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Apr 24 23:09:18.002712 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:18.002675    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Apr 24 23:09:19.247949 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.247905    2543 generic.go:358] "Generic (PLEG): container finished" podID="0355fe3d-1446-46b6-9804-132c10720573" containerID="7bc703361c62f13cd949b955ea5e8763a4a2afd624adf233777d5720b5017bd4" exitCode=0
Apr 24 23:09:19.248258 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.247919    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" event={"ID":"0355fe3d-1446-46b6-9804-132c10720573","Type":"ContainerDied","Data":"7bc703361c62f13cd949b955ea5e8763a4a2afd624adf233777d5720b5017bd4"}
Apr 24 23:09:19.287029 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.287009    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:09:19.353539 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.353515    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0355fe3d-1446-46b6-9804-132c10720573-proxy-tls\") pod \"0355fe3d-1446-46b6-9804-132c10720573\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") "
Apr 24 23:09:19.353656 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.353579    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0355fe3d-1446-46b6-9804-132c10720573-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"0355fe3d-1446-46b6-9804-132c10720573\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") "
Apr 24 23:09:19.353656 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.353629    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g67kk\" (UniqueName: \"kubernetes.io/projected/0355fe3d-1446-46b6-9804-132c10720573-kube-api-access-g67kk\") pod \"0355fe3d-1446-46b6-9804-132c10720573\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") "
Apr 24 23:09:19.353760 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.353688    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0355fe3d-1446-46b6-9804-132c10720573-kserve-provision-location\") pod \"0355fe3d-1446-46b6-9804-132c10720573\" (UID: \"0355fe3d-1446-46b6-9804-132c10720573\") "
Apr 24 23:09:19.354037 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.353915    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0355fe3d-1446-46b6-9804-132c10720573-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "0355fe3d-1446-46b6-9804-132c10720573" (UID: "0355fe3d-1446-46b6-9804-132c10720573"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:09:19.354122 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.354036    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0355fe3d-1446-46b6-9804-132c10720573-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0355fe3d-1446-46b6-9804-132c10720573" (UID: "0355fe3d-1446-46b6-9804-132c10720573"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:09:19.355587 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.355567    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0355fe3d-1446-46b6-9804-132c10720573-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0355fe3d-1446-46b6-9804-132c10720573" (UID: "0355fe3d-1446-46b6-9804-132c10720573"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:09:19.355776 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.355757    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0355fe3d-1446-46b6-9804-132c10720573-kube-api-access-g67kk" (OuterVolumeSpecName: "kube-api-access-g67kk") pod "0355fe3d-1446-46b6-9804-132c10720573" (UID: "0355fe3d-1446-46b6-9804-132c10720573"). InnerVolumeSpecName "kube-api-access-g67kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:09:19.454806 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.454781    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0355fe3d-1446-46b6-9804-132c10720573-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:09:19.454806 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.454805    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0355fe3d-1446-46b6-9804-132c10720573-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:09:19.454937 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.454815    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0355fe3d-1446-46b6-9804-132c10720573-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:09:19.454937 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:19.454825    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g67kk\" (UniqueName: \"kubernetes.io/projected/0355fe3d-1446-46b6-9804-132c10720573-kube-api-access-g67kk\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:09:20.253398 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:20.253364    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp" event={"ID":"0355fe3d-1446-46b6-9804-132c10720573","Type":"ContainerDied","Data":"a24fd4d91c3e7394b7471969a01f07840499966ac4c634d589a7b615452997b9"}
Apr 24 23:09:20.253823 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:20.253416    2543 scope.go:117] "RemoveContainer" containerID="457a43c860569c9dec32b39c29b01ac5cee374022bf78b65b3b7cd6b0e9a54ae"
Apr 24 23:09:20.253823 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:20.253373    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"
Apr 24 23:09:20.254771 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:20.254747    2543 generic.go:358] "Generic (PLEG): container finished" podID="e5df282a-0b72-430f-a9a8-740d68358161" containerID="78e86888f2ff648b9d4f8e68211d41dde845674110baae500520342d3599091a" exitCode=0
Apr 24 23:09:20.254861 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:20.254777    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" event={"ID":"e5df282a-0b72-430f-a9a8-740d68358161","Type":"ContainerDied","Data":"78e86888f2ff648b9d4f8e68211d41dde845674110baae500520342d3599091a"}
Apr 24 23:09:20.262914 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:20.262894    2543 scope.go:117] "RemoveContainer" containerID="7bc703361c62f13cd949b955ea5e8763a4a2afd624adf233777d5720b5017bd4"
Apr 24 23:09:20.269756 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:20.269738    2543 scope.go:117] "RemoveContainer" containerID="84d378b4ebccef9c24f0ba11b39d5d15ba9f153e8b18cacf3e8112cc31dbdb65"
Apr 24 23:09:20.285563 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:20.285543    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"]
Apr 24 23:09:20.289256 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:20.289238    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-p87zp"]
Apr 24 23:09:21.055689 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:21.055656    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0355fe3d-1446-46b6-9804-132c10720573" path="/var/lib/kubelet/pods/0355fe3d-1446-46b6-9804-132c10720573/volumes"
Apr 24 23:09:21.259109 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:21.259075    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" event={"ID":"e5df282a-0b72-430f-a9a8-740d68358161","Type":"ContainerStarted","Data":"6eafd6658a82e65d7e86f5aa45f25931cee8f780bddb263a02b78f3141146bb0"}
Apr 24 23:09:21.259109 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:21.259117    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" event={"ID":"e5df282a-0b72-430f-a9a8-740d68358161","Type":"ContainerStarted","Data":"35dd19eca690527983770fdaca0b91f62b05b0ca397b69f9415b5859ab1401cd"}
Apr 24 23:09:21.259642 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:21.259605    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:21.259698 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:21.259647    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:21.277300 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:21.277256    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" podStartSLOduration=6.277242891 podStartE2EDuration="6.277242891s" podCreationTimestamp="2026-04-24 23:09:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:09:21.274980353 +0000 UTC m=+2364.701377884" watchObservedRunningTime="2026-04-24 23:09:21.277242891 +0000 UTC m=+2364.703640421"
Apr 24 23:09:27.271092 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:27.271063    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:09:57.272166 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:57.272128    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.45:8080: connect: connection refused"
Apr 24 23:09:57.530944 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:57.530861    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:09:57.535565 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:09:57.535544    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:10:07.272062 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:07.271980    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.45:8080: connect: connection refused"
Apr 24 23:10:17.272077 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:17.272034    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.45:8080: connect: connection refused"
Apr 24 23:10:27.271968 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:27.271933    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.45:8080: connect: connection refused"
Apr 24 23:10:31.058697 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:31.058656    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:10:35.382660 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:35.382618    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"]
Apr 24 23:10:35.383046 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:35.382968    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kserve-container" containerID="cri-o://35dd19eca690527983770fdaca0b91f62b05b0ca397b69f9415b5859ab1401cd" gracePeriod=30
Apr 24 23:10:35.383046 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:35.383011    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kube-rbac-proxy" containerID="cri-o://6eafd6658a82e65d7e86f5aa45f25931cee8f780bddb263a02b78f3141146bb0" gracePeriod=30
Apr 24 23:10:36.487353 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:36.487306    2543 generic.go:358] "Generic (PLEG): container finished" podID="e5df282a-0b72-430f-a9a8-740d68358161" containerID="6eafd6658a82e65d7e86f5aa45f25931cee8f780bddb263a02b78f3141146bb0" exitCode=2
Apr 24 23:10:36.487695 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:36.487376    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" event={"ID":"e5df282a-0b72-430f-a9a8-740d68358161","Type":"ContainerDied","Data":"6eafd6658a82e65d7e86f5aa45f25931cee8f780bddb263a02b78f3141146bb0"}
Apr 24 23:10:37.265992 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.265949    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Apr 24 23:10:37.497483 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.497449    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"]
Apr 24 23:10:37.497825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.497766    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="storage-initializer"
Apr 24 23:10:37.497825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.497776    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="storage-initializer"
Apr 24 23:10:37.497825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.497786    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kserve-container"
Apr 24 23:10:37.497825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.497793    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kserve-container"
Apr 24 23:10:37.497825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.497804    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kube-rbac-proxy"
Apr 24 23:10:37.497825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.497809    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kube-rbac-proxy"
Apr 24 23:10:37.498014 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.497852    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kube-rbac-proxy"
Apr 24 23:10:37.498014 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.497863    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="0355fe3d-1446-46b6-9804-132c10720573" containerName="kserve-container"
Apr 24 23:10:37.500654 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.500638    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.503077 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.503059    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Apr 24 23:10:37.503200 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.503178    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Apr 24 23:10:37.511743 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.511720    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"]
Apr 24 23:10:37.609063 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.608984    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b76c5d-1762-4521-b65b-47fba03e5520-proxy-tls\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.609063 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.609044    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xbz\" (UniqueName: \"kubernetes.io/projected/46b76c5d-1762-4521-b65b-47fba03e5520-kube-api-access-k9xbz\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.609224 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.609077    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b76c5d-1762-4521-b65b-47fba03e5520-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.609224 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.609165    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b76c5d-1762-4521-b65b-47fba03e5520-kserve-provision-location\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.709906 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.709870    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b76c5d-1762-4521-b65b-47fba03e5520-proxy-tls\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.709906 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.709911    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xbz\" (UniqueName: \"kubernetes.io/projected/46b76c5d-1762-4521-b65b-47fba03e5520-kube-api-access-k9xbz\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.710152 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.709933    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b76c5d-1762-4521-b65b-47fba03e5520-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.710152 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.709983    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b76c5d-1762-4521-b65b-47fba03e5520-kserve-provision-location\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.710426 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.710401    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b76c5d-1762-4521-b65b-47fba03e5520-kserve-provision-location\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.710717 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.710701    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b76c5d-1762-4521-b65b-47fba03e5520-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.712664 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.712643    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b76c5d-1762-4521-b65b-47fba03e5520-proxy-tls\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.717939 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.717904    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xbz\" (UniqueName: \"kubernetes.io/projected/46b76c5d-1762-4521-b65b-47fba03e5520-kube-api-access-k9xbz\") pod \"isvc-sklearn-predictor-5cf96b68d-vjw9m\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.810803 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.810779    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:37.928083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.928060    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"]
Apr 24 23:10:37.930462 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:10:37.930432    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b76c5d_1762_4521_b65b_47fba03e5520.slice/crio-ec648e8c136729f98a993bd6e345f9198b760ac5bb0066e8cd3f3cd1276c7d3a WatchSource:0}: Error finding container ec648e8c136729f98a993bd6e345f9198b760ac5bb0066e8cd3f3cd1276c7d3a: Status 404 returned error can't find the container with id ec648e8c136729f98a993bd6e345f9198b760ac5bb0066e8cd3f3cd1276c7d3a
Apr 24 23:10:37.932828 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:37.932812    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 23:10:38.494613 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:38.494578    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" event={"ID":"46b76c5d-1762-4521-b65b-47fba03e5520","Type":"ContainerStarted","Data":"7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43"}
Apr 24 23:10:38.494613 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:38.494615    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" event={"ID":"46b76c5d-1762-4521-b65b-47fba03e5520","Type":"ContainerStarted","Data":"ec648e8c136729f98a993bd6e345f9198b760ac5bb0066e8cd3f3cd1276c7d3a"}
Apr 24 23:10:40.507180 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.507148    2543 generic.go:358] "Generic (PLEG): container finished" podID="e5df282a-0b72-430f-a9a8-740d68358161" containerID="35dd19eca690527983770fdaca0b91f62b05b0ca397b69f9415b5859ab1401cd" exitCode=0
Apr 24 23:10:40.507550 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.507217    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" event={"ID":"e5df282a-0b72-430f-a9a8-740d68358161","Type":"ContainerDied","Data":"35dd19eca690527983770fdaca0b91f62b05b0ca397b69f9415b5859ab1401cd"}
Apr 24 23:10:40.721159 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.721135    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:10:40.830586 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.830514    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5df282a-0b72-430f-a9a8-740d68358161-proxy-tls\") pod \"e5df282a-0b72-430f-a9a8-740d68358161\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") "
Apr 24 23:10:40.830586 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.830552    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5df282a-0b72-430f-a9a8-740d68358161-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"e5df282a-0b72-430f-a9a8-740d68358161\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") "
Apr 24 23:10:40.830802 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.830605    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k7p4\" (UniqueName: \"kubernetes.io/projected/e5df282a-0b72-430f-a9a8-740d68358161-kube-api-access-5k7p4\") pod \"e5df282a-0b72-430f-a9a8-740d68358161\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") "
Apr 24 23:10:40.830802 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.830646    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5df282a-0b72-430f-a9a8-740d68358161-kserve-provision-location\") pod \"e5df282a-0b72-430f-a9a8-740d68358161\" (UID: \"e5df282a-0b72-430f-a9a8-740d68358161\") "
Apr 24 23:10:40.830995 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.830962    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5df282a-0b72-430f-a9a8-740d68358161-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5df282a-0b72-430f-a9a8-740d68358161" (UID: "e5df282a-0b72-430f-a9a8-740d68358161"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:10:40.831051 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.830965    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5df282a-0b72-430f-a9a8-740d68358161-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "e5df282a-0b72-430f-a9a8-740d68358161" (UID: "e5df282a-0b72-430f-a9a8-740d68358161"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:10:40.832630 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.832610    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5df282a-0b72-430f-a9a8-740d68358161-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5df282a-0b72-430f-a9a8-740d68358161" (UID: "e5df282a-0b72-430f-a9a8-740d68358161"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:10:40.832764 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.832744    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5df282a-0b72-430f-a9a8-740d68358161-kube-api-access-5k7p4" (OuterVolumeSpecName: "kube-api-access-5k7p4") pod "e5df282a-0b72-430f-a9a8-740d68358161" (UID: "e5df282a-0b72-430f-a9a8-740d68358161"). InnerVolumeSpecName "kube-api-access-5k7p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:10:40.931113 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.931087    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5df282a-0b72-430f-a9a8-740d68358161-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:10:40.931113 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.931109    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k7p4\" (UniqueName: \"kubernetes.io/projected/e5df282a-0b72-430f-a9a8-740d68358161-kube-api-access-5k7p4\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:10:40.931248 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.931121    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5df282a-0b72-430f-a9a8-740d68358161-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:10:40.931248 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:40.931130    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5df282a-0b72-430f-a9a8-740d68358161-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:10:41.511788 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:41.511759    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2" event={"ID":"e5df282a-0b72-430f-a9a8-740d68358161","Type":"ContainerDied","Data":"9983652f7bd2cec0d00d0fda3cdaaef7632337bd9a3c771a7353dc609e0106ae"}
Apr 24 23:10:41.512199 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:41.511806    2543 scope.go:117] "RemoveContainer" containerID="6eafd6658a82e65d7e86f5aa45f25931cee8f780bddb263a02b78f3141146bb0"
Apr 24 23:10:41.512199 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:41.511822    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"
Apr 24 23:10:41.513173 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:41.513143    2543 generic.go:358] "Generic (PLEG): container finished" podID="46b76c5d-1762-4521-b65b-47fba03e5520" containerID="7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43" exitCode=0
Apr 24 23:10:41.513270 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:41.513244    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" event={"ID":"46b76c5d-1762-4521-b65b-47fba03e5520","Type":"ContainerDied","Data":"7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43"}
Apr 24 23:10:41.519830 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:41.519815    2543 scope.go:117] "RemoveContainer" containerID="35dd19eca690527983770fdaca0b91f62b05b0ca397b69f9415b5859ab1401cd"
Apr 24 23:10:41.526878 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:41.526860    2543 scope.go:117] "RemoveContainer" containerID="78e86888f2ff648b9d4f8e68211d41dde845674110baae500520342d3599091a"
Apr 24 23:10:41.530754 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:41.530735    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"]
Apr 24 23:10:41.533914 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:41.533892    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-vszp2"]
Apr 24 23:10:42.518159 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:42.518116    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" event={"ID":"46b76c5d-1762-4521-b65b-47fba03e5520","Type":"ContainerStarted","Data":"4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140"}
Apr 24 23:10:42.518159 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:42.518149    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" event={"ID":"46b76c5d-1762-4521-b65b-47fba03e5520","Type":"ContainerStarted","Data":"afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5"}
Apr 24 23:10:42.518631 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:42.518435    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:42.538684 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:42.538642    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podStartSLOduration=5.538627329 podStartE2EDuration="5.538627329s" podCreationTimestamp="2026-04-24 23:10:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:10:42.536428144 +0000 UTC m=+2445.962825671" watchObservedRunningTime="2026-04-24 23:10:42.538627329 +0000 UTC m=+2445.965024892"
Apr 24 23:10:43.055637 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:43.055605    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5df282a-0b72-430f-a9a8-740d68358161" path="/var/lib/kubelet/pods/e5df282a-0b72-430f-a9a8-740d68358161/volumes"
Apr 24 23:10:43.520727 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:43.520691    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:43.521759 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:43.521735    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Apr 24 23:10:44.523288 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:44.523250    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Apr 24 23:10:49.527459 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:49.527423    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:10:49.528037 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:49.528011    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Apr 24 23:10:59.528257 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:10:59.528220    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Apr 24 23:11:09.528701 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:09.528662    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Apr 24 23:11:19.528260 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:19.528220    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Apr 24 23:11:29.528906 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:29.528866    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Apr 24 23:11:39.528537 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:39.528454    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Apr 24 23:11:49.528521 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:49.528495    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:11:57.588611 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.588579    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"]
Apr 24 23:11:57.589063 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.589007    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" containerID="cri-o://afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5" gracePeriod=30
Apr 24 23:11:57.589063 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.589030    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kube-rbac-proxy" containerID="cri-o://4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140" gracePeriod=30
Apr 24 23:11:57.667113 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.667090    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"]
Apr 24 23:11:57.667418 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.667390    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="storage-initializer"
Apr 24 23:11:57.667480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.667420    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="storage-initializer"
Apr 24 23:11:57.667480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.667431    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kube-rbac-proxy"
Apr 24 23:11:57.667480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.667445    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kube-rbac-proxy"
Apr 24 23:11:57.667480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.667458    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kserve-container"
Apr 24 23:11:57.667480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.667464    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kserve-container"
Apr 24 23:11:57.667638 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.667511    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kserve-container"
Apr 24 23:11:57.667638 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.667520    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="e5df282a-0b72-430f-a9a8-740d68358161" containerName="kube-rbac-proxy"
Apr 24 23:11:57.670401 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.670386    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.672879 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.672861    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Apr 24 23:11:57.672952 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.672867    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Apr 24 23:11:57.677967 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.677948    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"]
Apr 24 23:11:57.680737 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.680714    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvp42\" (UniqueName: \"kubernetes.io/projected/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kube-api-access-wvp42\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.680824 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.680741    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.680870 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.680843    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/900d0e39-350c-44e3-a16f-6eaa4e0e8209-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.680906 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.680878    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/900d0e39-350c-44e3-a16f-6eaa4e0e8209-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.731468 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.731432    2543 generic.go:358] "Generic (PLEG): container finished" podID="46b76c5d-1762-4521-b65b-47fba03e5520" containerID="4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140" exitCode=2
Apr 24 23:11:57.731574 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.731496    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" event={"ID":"46b76c5d-1762-4521-b65b-47fba03e5520","Type":"ContainerDied","Data":"4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140"}
Apr 24 23:11:57.781364 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.781312    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/900d0e39-350c-44e3-a16f-6eaa4e0e8209-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.781483 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.781397    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wvp42\" (UniqueName: \"kubernetes.io/projected/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kube-api-access-wvp42\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.781483 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.781424    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.781588 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.781507    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/900d0e39-350c-44e3-a16f-6eaa4e0e8209-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.781665 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:11:57.781644    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Apr 24 23:11:57.781746 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:11:57.781735    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/900d0e39-350c-44e3-a16f-6eaa4e0e8209-proxy-tls podName:900d0e39-350c-44e3-a16f-6eaa4e0e8209 nodeName:}" failed. No retries permitted until 2026-04-24 23:11:58.281711368 +0000 UTC m=+2521.708108882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/900d0e39-350c-44e3-a16f-6eaa4e0e8209-proxy-tls") pod "sklearn-v2-mlserver-predictor-65d8664766-jhmvx" (UID: "900d0e39-350c-44e3-a16f-6eaa4e0e8209") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Apr 24 23:11:57.781790 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.781750    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.782034 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.782017    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/900d0e39-350c-44e3-a16f-6eaa4e0e8209-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:57.789831 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:57.789808    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvp42\" (UniqueName: \"kubernetes.io/projected/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kube-api-access-wvp42\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:58.286597 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:58.286558    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/900d0e39-350c-44e3-a16f-6eaa4e0e8209-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:58.289035 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:58.289013    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/900d0e39-350c-44e3-a16f-6eaa4e0e8209-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-jhmvx\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:58.581529 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:58.581450    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:11:58.697275 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:58.697243    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"]
Apr 24 23:11:58.700100 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:11:58.700063    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900d0e39_350c_44e3_a16f_6eaa4e0e8209.slice/crio-e12bd71a73e2e46bb1605060bfb928bfecc290d4ec019491c537844b4758bdf0 WatchSource:0}: Error finding container e12bd71a73e2e46bb1605060bfb928bfecc290d4ec019491c537844b4758bdf0: Status 404 returned error can't find the container with id e12bd71a73e2e46bb1605060bfb928bfecc290d4ec019491c537844b4758bdf0
Apr 24 23:11:58.736081 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:58.736057    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" event={"ID":"900d0e39-350c-44e3-a16f-6eaa4e0e8209","Type":"ContainerStarted","Data":"e12bd71a73e2e46bb1605060bfb928bfecc290d4ec019491c537844b4758bdf0"}
Apr 24 23:11:59.524485 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:59.524442    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Apr 24 23:11:59.528829 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:59.528798    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Apr 24 23:11:59.740480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:11:59.740450    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" event={"ID":"900d0e39-350c-44e3-a16f-6eaa4e0e8209","Type":"ContainerStarted","Data":"558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d"}
Apr 24 23:12:01.328312 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.328291    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:12:01.409427 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.409355    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b76c5d-1762-4521-b65b-47fba03e5520-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"46b76c5d-1762-4521-b65b-47fba03e5520\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") "
Apr 24 23:12:01.409427 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.409409    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b76c5d-1762-4521-b65b-47fba03e5520-kserve-provision-location\") pod \"46b76c5d-1762-4521-b65b-47fba03e5520\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") "
Apr 24 23:12:01.409592 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.409447    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9xbz\" (UniqueName: \"kubernetes.io/projected/46b76c5d-1762-4521-b65b-47fba03e5520-kube-api-access-k9xbz\") pod \"46b76c5d-1762-4521-b65b-47fba03e5520\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") "
Apr 24 23:12:01.409592 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.409471    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b76c5d-1762-4521-b65b-47fba03e5520-proxy-tls\") pod \"46b76c5d-1762-4521-b65b-47fba03e5520\" (UID: \"46b76c5d-1762-4521-b65b-47fba03e5520\") "
Apr 24 23:12:01.409766 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.409737    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b76c5d-1762-4521-b65b-47fba03e5520-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "46b76c5d-1762-4521-b65b-47fba03e5520" (UID: "46b76c5d-1762-4521-b65b-47fba03e5520"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:12:01.409835 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.409762    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b76c5d-1762-4521-b65b-47fba03e5520-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46b76c5d-1762-4521-b65b-47fba03e5520" (UID: "46b76c5d-1762-4521-b65b-47fba03e5520"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:12:01.411586 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.411559    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b76c5d-1762-4521-b65b-47fba03e5520-kube-api-access-k9xbz" (OuterVolumeSpecName: "kube-api-access-k9xbz") pod "46b76c5d-1762-4521-b65b-47fba03e5520" (UID: "46b76c5d-1762-4521-b65b-47fba03e5520"). InnerVolumeSpecName "kube-api-access-k9xbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:12:01.411686 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.411609    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b76c5d-1762-4521-b65b-47fba03e5520-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46b76c5d-1762-4521-b65b-47fba03e5520" (UID: "46b76c5d-1762-4521-b65b-47fba03e5520"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:12:01.510645 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.510618    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46b76c5d-1762-4521-b65b-47fba03e5520-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:12:01.510645 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.510645    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46b76c5d-1762-4521-b65b-47fba03e5520-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:12:01.510798 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.510656    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xbz\" (UniqueName: \"kubernetes.io/projected/46b76c5d-1762-4521-b65b-47fba03e5520-kube-api-access-k9xbz\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:12:01.510798 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.510665    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46b76c5d-1762-4521-b65b-47fba03e5520-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:12:01.747262 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.747228    2543 generic.go:358] "Generic (PLEG): container finished" podID="46b76c5d-1762-4521-b65b-47fba03e5520" containerID="afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5" exitCode=0
Apr 24 23:12:01.747420 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.747281    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" event={"ID":"46b76c5d-1762-4521-b65b-47fba03e5520","Type":"ContainerDied","Data":"afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5"}
Apr 24 23:12:01.747420 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.747314    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m" event={"ID":"46b76c5d-1762-4521-b65b-47fba03e5520","Type":"ContainerDied","Data":"ec648e8c136729f98a993bd6e345f9198b760ac5bb0066e8cd3f3cd1276c7d3a"}
Apr 24 23:12:01.747420 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.747316    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"
Apr 24 23:12:01.747420 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.747329    2543 scope.go:117] "RemoveContainer" containerID="4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140"
Apr 24 23:12:01.760502 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.760483    2543 scope.go:117] "RemoveContainer" containerID="afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5"
Apr 24 23:12:01.767621 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.767605    2543 scope.go:117] "RemoveContainer" containerID="7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43"
Apr 24 23:12:01.772509 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.772484    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"]
Apr 24 23:12:01.774520 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.774505    2543 scope.go:117] "RemoveContainer" containerID="4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140"
Apr 24 23:12:01.774760 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:12:01.774739    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140\": container with ID starting with 4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140 not found: ID does not exist" containerID="4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140"
Apr 24 23:12:01.774832 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.774766    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140"} err="failed to get container status \"4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140\": rpc error: code = NotFound desc = could not find container \"4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140\": container with ID starting with 4b4354a741911b63e3c578c4ab2ef88fa9647700a798d09f502d3a3c02693140 not found: ID does not exist"
Apr 24 23:12:01.774832 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.774783    2543 scope.go:117] "RemoveContainer" containerID="afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5"
Apr 24 23:12:01.775255 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:12:01.775064    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5\": container with ID starting with afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5 not found: ID does not exist" containerID="afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5"
Apr 24 23:12:01.775255 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.775139    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5"} err="failed to get container status \"afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5\": rpc error: code = NotFound desc = could not find container \"afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5\": container with ID starting with afc7492be4530062fe8bbe129d3beb06e86dca8973ed00bfb8a297d946a5cbc5 not found: ID does not exist"
Apr 24 23:12:01.775255 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.775160    2543 scope.go:117] "RemoveContainer" containerID="7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43"
Apr 24 23:12:01.775643 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:12:01.775622    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43\": container with ID starting with 7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43 not found: ID does not exist" containerID="7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43"
Apr 24 23:12:01.775715 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.775649    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43"} err="failed to get container status \"7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43\": rpc error: code = NotFound desc = could not find container \"7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43\": container with ID starting with 7c340a2d42244e56709d8f833e2eec9fde84686cadc82e751e62770648f93f43 not found: ID does not exist"
Apr 24 23:12:01.777429 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:01.777410    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5cf96b68d-vjw9m"]
Apr 24 23:12:02.752188 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:02.752155    2543 generic.go:358] "Generic (PLEG): container finished" podID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerID="558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d" exitCode=0
Apr 24 23:12:02.752601 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:02.752218    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" event={"ID":"900d0e39-350c-44e3-a16f-6eaa4e0e8209","Type":"ContainerDied","Data":"558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d"}
Apr 24 23:12:03.054732 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:03.054660    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" path="/var/lib/kubelet/pods/46b76c5d-1762-4521-b65b-47fba03e5520/volumes"
Apr 24 23:12:03.757250 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:03.757214    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" event={"ID":"900d0e39-350c-44e3-a16f-6eaa4e0e8209","Type":"ContainerStarted","Data":"0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41"}
Apr 24 23:12:03.757250 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:03.757248    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" event={"ID":"900d0e39-350c-44e3-a16f-6eaa4e0e8209","Type":"ContainerStarted","Data":"26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636"}
Apr 24 23:12:03.757764 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:03.757472    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:12:03.776725 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:03.776685    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" podStartSLOduration=6.7766703150000005 podStartE2EDuration="6.776670315s" podCreationTimestamp="2026-04-24 23:11:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:12:03.775526771 +0000 UTC m=+2527.201924303" watchObservedRunningTime="2026-04-24 23:12:03.776670315 +0000 UTC m=+2527.203068254"
Apr 24 23:12:04.760633 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:04.760606    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:12:10.768386 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:10.768360    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:12:40.811600 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:40.811561    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Apr 24 23:12:50.771501 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:50.771473    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:12:57.770740 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.770708    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"]
Apr 24 23:12:57.771185 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.771017    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kserve-container" containerID="cri-o://26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636" gracePeriod=30
Apr 24 23:12:57.771185 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.771052    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kube-rbac-proxy" containerID="cri-o://0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41" gracePeriod=30
Apr 24 23:12:57.843511 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.843458    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"]
Apr 24 23:12:57.843762 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.843749    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="storage-initializer"
Apr 24 23:12:57.843821 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.843763    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="storage-initializer"
Apr 24 23:12:57.843821 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.843772    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container"
Apr 24 23:12:57.843821 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.843777    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container"
Apr 24 23:12:57.843821 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.843793    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kube-rbac-proxy"
Apr 24 23:12:57.843821 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.843799    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kube-rbac-proxy"
Apr 24 23:12:57.844007 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.843841    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kserve-container"
Apr 24 23:12:57.844007 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.843850    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="46b76c5d-1762-4521-b65b-47fba03e5520" containerName="kube-rbac-proxy"
Apr 24 23:12:57.846827 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.846813    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:57.849778 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.849759    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Apr 24 23:12:57.849864 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.849782    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Apr 24 23:12:57.856708 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.856687    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"]
Apr 24 23:12:57.905380 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.905357    2543 generic.go:358] "Generic (PLEG): container finished" podID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerID="0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41" exitCode=2
Apr 24 23:12:57.905472 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:57.905436    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" event={"ID":"900d0e39-350c-44e3-a16f-6eaa4e0e8209","Type":"ContainerDied","Data":"0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41"}
Apr 24 23:12:58.024406 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.024324    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b0a14a-b549-4724-8c64-7ede4ab92864-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.024406 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.024388    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b0a14a-b549-4724-8c64-7ede4ab92864-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.024538 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.024420    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b0a14a-b549-4724-8c64-7ede4ab92864-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.024538 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.024460    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrtl\" (UniqueName: \"kubernetes.io/projected/88b0a14a-b549-4724-8c64-7ede4ab92864-kube-api-access-qwrtl\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.125722 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.125697    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b0a14a-b549-4724-8c64-7ede4ab92864-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.125812 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.125739    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrtl\" (UniqueName: \"kubernetes.io/projected/88b0a14a-b549-4724-8c64-7ede4ab92864-kube-api-access-qwrtl\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.125812 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.125789    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b0a14a-b549-4724-8c64-7ede4ab92864-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.125886 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.125828    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b0a14a-b549-4724-8c64-7ede4ab92864-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.125966 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:12:58.125948    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Apr 24 23:12:58.126037 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:12:58.126025    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88b0a14a-b549-4724-8c64-7ede4ab92864-proxy-tls podName:88b0a14a-b549-4724-8c64-7ede4ab92864 nodeName:}" failed. No retries permitted until 2026-04-24 23:12:58.626006884 +0000 UTC m=+2582.052404395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/88b0a14a-b549-4724-8c64-7ede4ab92864-proxy-tls") pod "isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" (UID: "88b0a14a-b549-4724-8c64-7ede4ab92864") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Apr 24 23:12:58.126127 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.126110    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b0a14a-b549-4724-8c64-7ede4ab92864-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.126369 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.126324    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b0a14a-b549-4724-8c64-7ede4ab92864-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.134714 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.134690    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrtl\" (UniqueName: \"kubernetes.io/projected/88b0a14a-b549-4724-8c64-7ede4ab92864-kube-api-access-qwrtl\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.630503 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.630389    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b0a14a-b549-4724-8c64-7ede4ab92864-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.632947 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.632914    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b0a14a-b549-4724-8c64-7ede4ab92864-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.756086 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.756056    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:12:58.872239 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.872215    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"]
Apr 24 23:12:58.874120 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:12:58.874093    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b0a14a_b549_4724_8c64_7ede4ab92864.slice/crio-24b9dd7b8805a6aef55ca5491b3f4ae585c03283173f03f9d78639454e98b73b WatchSource:0}: Error finding container 24b9dd7b8805a6aef55ca5491b3f4ae585c03283173f03f9d78639454e98b73b: Status 404 returned error can't find the container with id 24b9dd7b8805a6aef55ca5491b3f4ae585c03283173f03f9d78639454e98b73b
Apr 24 23:12:58.910081 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:58.910042    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" event={"ID":"88b0a14a-b549-4724-8c64-7ede4ab92864","Type":"ContainerStarted","Data":"24b9dd7b8805a6aef55ca5491b3f4ae585c03283173f03f9d78639454e98b73b"}
Apr 24 23:12:59.914725 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:12:59.914642    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" event={"ID":"88b0a14a-b549-4724-8c64-7ede4ab92864","Type":"ContainerStarted","Data":"48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7"}
Apr 24 23:13:00.764608 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:00.764568    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Apr 24 23:13:04.706482 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.706457    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:13:04.775527 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.775470    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kserve-provision-location\") pod \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") "
Apr 24 23:13:04.775643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.775526    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/900d0e39-350c-44e3-a16f-6eaa4e0e8209-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") "
Apr 24 23:13:04.775643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.775563    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvp42\" (UniqueName: \"kubernetes.io/projected/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kube-api-access-wvp42\") pod \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") "
Apr 24 23:13:04.775643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.775596    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/900d0e39-350c-44e3-a16f-6eaa4e0e8209-proxy-tls\") pod \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\" (UID: \"900d0e39-350c-44e3-a16f-6eaa4e0e8209\") "
Apr 24 23:13:04.775802 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.775777    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "900d0e39-350c-44e3-a16f-6eaa4e0e8209" (UID: "900d0e39-350c-44e3-a16f-6eaa4e0e8209"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:13:04.775899 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.775881    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900d0e39-350c-44e3-a16f-6eaa4e0e8209-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "900d0e39-350c-44e3-a16f-6eaa4e0e8209" (UID: "900d0e39-350c-44e3-a16f-6eaa4e0e8209"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:13:04.777623 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.777602    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kube-api-access-wvp42" (OuterVolumeSpecName: "kube-api-access-wvp42") pod "900d0e39-350c-44e3-a16f-6eaa4e0e8209" (UID: "900d0e39-350c-44e3-a16f-6eaa4e0e8209"). InnerVolumeSpecName "kube-api-access-wvp42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:13:04.777725 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.777705    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900d0e39-350c-44e3-a16f-6eaa4e0e8209-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "900d0e39-350c-44e3-a16f-6eaa4e0e8209" (UID: "900d0e39-350c-44e3-a16f-6eaa4e0e8209"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:13:04.876581 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.876562    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/900d0e39-350c-44e3-a16f-6eaa4e0e8209-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:13:04.876581 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.876582    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:13:04.876717 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.876592    2543 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/900d0e39-350c-44e3-a16f-6eaa4e0e8209-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:13:04.876717 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.876601    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvp42\" (UniqueName: \"kubernetes.io/projected/900d0e39-350c-44e3-a16f-6eaa4e0e8209-kube-api-access-wvp42\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:13:04.929478 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.929455    2543 generic.go:358] "Generic (PLEG): container finished" podID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerID="48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7" exitCode=0
Apr 24 23:13:04.929584 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.929534    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" event={"ID":"88b0a14a-b549-4724-8c64-7ede4ab92864","Type":"ContainerDied","Data":"48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7"}
Apr 24 23:13:04.931123 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.931103    2543 generic.go:358] "Generic (PLEG): container finished" podID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerID="26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636" exitCode=0
Apr 24 23:13:04.931209 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.931151    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" event={"ID":"900d0e39-350c-44e3-a16f-6eaa4e0e8209","Type":"ContainerDied","Data":"26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636"}
Apr 24 23:13:04.931209 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.931182    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx" event={"ID":"900d0e39-350c-44e3-a16f-6eaa4e0e8209","Type":"ContainerDied","Data":"e12bd71a73e2e46bb1605060bfb928bfecc290d4ec019491c537844b4758bdf0"}
Apr 24 23:13:04.931209 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.931201    2543 scope.go:117] "RemoveContainer" containerID="0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41"
Apr 24 23:13:04.931306 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.931278    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"
Apr 24 23:13:04.941592 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.941563    2543 scope.go:117] "RemoveContainer" containerID="26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636"
Apr 24 23:13:04.948928 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.948911    2543 scope.go:117] "RemoveContainer" containerID="558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d"
Apr 24 23:13:04.956304 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.956290    2543 scope.go:117] "RemoveContainer" containerID="0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41"
Apr 24 23:13:04.956591 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:13:04.956572    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41\": container with ID starting with 0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41 not found: ID does not exist" containerID="0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41"
Apr 24 23:13:04.956688 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.956603    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41"} err="failed to get container status \"0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41\": rpc error: code = NotFound desc = could not find container \"0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41\": container with ID starting with 0fbd0f77f5e0ee427ba15ae34bc38d990bbfd118d2b814a5a97147dce5748e41 not found: ID does not exist"
Apr 24 23:13:04.956688 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.956621    2543 scope.go:117] "RemoveContainer" containerID="26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636"
Apr 24 23:13:04.956892 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:13:04.956873    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636\": container with ID starting with 26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636 not found: ID does not exist" containerID="26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636"
Apr 24 23:13:04.957011 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.956895    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636"} err="failed to get container status \"26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636\": rpc error: code = NotFound desc = could not find container \"26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636\": container with ID starting with 26f728fbc9bc283689885e9ab8ceacc7cf6f03232f4d23cfd5bbc0bc6bad3636 not found: ID does not exist"
Apr 24 23:13:04.957011 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.956917    2543 scope.go:117] "RemoveContainer" containerID="558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d"
Apr 24 23:13:04.957279 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:13:04.957249    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d\": container with ID starting with 558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d not found: ID does not exist" containerID="558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d"
Apr 24 23:13:04.957478 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.957287    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d"} err="failed to get container status \"558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d\": rpc error: code = NotFound desc = could not find container \"558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d\": container with ID starting with 558d70e6f9232e555c5197f5acb211f7fc48e7cee5ee069b7d2f6199360c074d not found: ID does not exist"
Apr 24 23:13:04.963796 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.963769    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"]
Apr 24 23:13:04.963796 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:04.963813    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-jhmvx"]
Apr 24 23:13:05.055850 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:05.055827    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" path="/var/lib/kubelet/pods/900d0e39-350c-44e3-a16f-6eaa4e0e8209/volumes"
Apr 24 23:13:05.935575 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:05.935538    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" event={"ID":"88b0a14a-b549-4724-8c64-7ede4ab92864","Type":"ContainerStarted","Data":"2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a"}
Apr 24 23:13:05.935575 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:05.935580    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" event={"ID":"88b0a14a-b549-4724-8c64-7ede4ab92864","Type":"ContainerStarted","Data":"86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59"}
Apr 24 23:13:05.936107 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:05.935823    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:13:05.936107 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:05.935934    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:13:05.937459 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:05.937435    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Apr 24 23:13:05.955371 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:05.955310    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" podStartSLOduration=8.955298157 podStartE2EDuration="8.955298157s" podCreationTimestamp="2026-04-24 23:12:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:13:05.954485846 +0000 UTC m=+2589.380883376" watchObservedRunningTime="2026-04-24 23:13:05.955298157 +0000 UTC m=+2589.381695699"
Apr 24 23:13:06.939370 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:06.939263    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Apr 24 23:13:11.943650 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:11.943624    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:13:11.944118 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:11.944093    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Apr 24 23:13:21.944590 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:21.944561    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:13:34.755157 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.755127    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw_88b0a14a-b549-4724-8c64-7ede4ab92864/kserve-container/0.log"
Apr 24 23:13:34.884437 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.884374    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"]
Apr 24 23:13:34.885036 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.885004    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kserve-container" containerID="cri-o://86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59" gracePeriod=30
Apr 24 23:13:34.885286 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.885093    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kube-rbac-proxy" containerID="cri-o://2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a" gracePeriod=30
Apr 24 23:13:34.957282 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.957246    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"]
Apr 24 23:13:34.957571 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.957558    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kserve-container"
Apr 24 23:13:34.957630 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.957572    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kserve-container"
Apr 24 23:13:34.957630 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.957589    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kube-rbac-proxy"
Apr 24 23:13:34.957630 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.957595    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kube-rbac-proxy"
Apr 24 23:13:34.957630 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.957605    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="storage-initializer"
Apr 24 23:13:34.957630 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.957611    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="storage-initializer"
Apr 24 23:13:34.957809 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.957657    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kube-rbac-proxy"
Apr 24 23:13:34.957809 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.957666    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="900d0e39-350c-44e3-a16f-6eaa4e0e8209" containerName="kserve-container"
Apr 24 23:13:34.963483 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.961765    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:34.964585 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.964562    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Apr 24 23:13:34.964684 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.964607    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Apr 24 23:13:34.971656 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:34.971629    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"]
Apr 24 23:13:35.021210 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.021151    2543 generic.go:358] "Generic (PLEG): container finished" podID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerID="2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a" exitCode=2
Apr 24 23:13:35.021210 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.021199    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" event={"ID":"88b0a14a-b549-4724-8c64-7ede4ab92864","Type":"ContainerDied","Data":"2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a"}
Apr 24 23:13:35.105027 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.104990    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea16570-90f9-4f35-be85-00593496a913-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.105189 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.105041    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea16570-90f9-4f35-be85-00593496a913-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.105189 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.105112    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzw9q\" (UniqueName: \"kubernetes.io/projected/2ea16570-90f9-4f35-be85-00593496a913-kube-api-access-jzw9q\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.105189 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.105156    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea16570-90f9-4f35-be85-00593496a913-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.206353 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.206302    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jzw9q\" (UniqueName: \"kubernetes.io/projected/2ea16570-90f9-4f35-be85-00593496a913-kube-api-access-jzw9q\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.206507 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.206362    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea16570-90f9-4f35-be85-00593496a913-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.206507 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.206426    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea16570-90f9-4f35-be85-00593496a913-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.206507 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.206453    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea16570-90f9-4f35-be85-00593496a913-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.206645 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:13:35.206572    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Apr 24 23:13:35.206689 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:13:35.206649    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea16570-90f9-4f35-be85-00593496a913-proxy-tls podName:2ea16570-90f9-4f35-be85-00593496a913 nodeName:}" failed. No retries permitted until 2026-04-24 23:13:35.706628498 +0000 UTC m=+2619.133026005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2ea16570-90f9-4f35-be85-00593496a913-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" (UID: "2ea16570-90f9-4f35-be85-00593496a913") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Apr 24 23:13:35.206803 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.206781    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea16570-90f9-4f35-be85-00593496a913-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.207037 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.207017    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea16570-90f9-4f35-be85-00593496a913-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.217581 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.217553    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzw9q\" (UniqueName: \"kubernetes.io/projected/2ea16570-90f9-4f35-be85-00593496a913-kube-api-access-jzw9q\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.709842 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.709809    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea16570-90f9-4f35-be85-00593496a913-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.712055 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.712031    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea16570-90f9-4f35-be85-00593496a913-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.820263 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.820243    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:13:35.874524 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.874454    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:35.911479 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.911450    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b0a14a-b549-4724-8c64-7ede4ab92864-proxy-tls\") pod \"88b0a14a-b549-4724-8c64-7ede4ab92864\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") "
Apr 24 23:13:35.911666 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.911488    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwrtl\" (UniqueName: \"kubernetes.io/projected/88b0a14a-b549-4724-8c64-7ede4ab92864-kube-api-access-qwrtl\") pod \"88b0a14a-b549-4724-8c64-7ede4ab92864\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") "
Apr 24 23:13:35.911666 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.911519    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b0a14a-b549-4724-8c64-7ede4ab92864-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"88b0a14a-b549-4724-8c64-7ede4ab92864\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") "
Apr 24 23:13:35.911666 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.911566    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b0a14a-b549-4724-8c64-7ede4ab92864-kserve-provision-location\") pod \"88b0a14a-b549-4724-8c64-7ede4ab92864\" (UID: \"88b0a14a-b549-4724-8c64-7ede4ab92864\") "
Apr 24 23:13:35.912526 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.912081    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b0a14a-b549-4724-8c64-7ede4ab92864-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "88b0a14a-b549-4724-8c64-7ede4ab92864" (UID: "88b0a14a-b549-4724-8c64-7ede4ab92864"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:13:35.913854 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.913823    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b0a14a-b549-4724-8c64-7ede4ab92864-kube-api-access-qwrtl" (OuterVolumeSpecName: "kube-api-access-qwrtl") pod "88b0a14a-b549-4724-8c64-7ede4ab92864" (UID: "88b0a14a-b549-4724-8c64-7ede4ab92864"). InnerVolumeSpecName "kube-api-access-qwrtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:13:35.914019 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.913994    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b0a14a-b549-4724-8c64-7ede4ab92864-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88b0a14a-b549-4724-8c64-7ede4ab92864" (UID: "88b0a14a-b549-4724-8c64-7ede4ab92864"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:13:35.918621 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:35.918592    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b0a14a-b549-4724-8c64-7ede4ab92864-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88b0a14a-b549-4724-8c64-7ede4ab92864" (UID: "88b0a14a-b549-4724-8c64-7ede4ab92864"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:13:36.002405 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.002362    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"]
Apr 24 23:13:36.004593 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:13:36.004564    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea16570_90f9_4f35_be85_00593496a913.slice/crio-202091dc883856e6d9780dee48f162a37fcafa1775484696235b4ea2699b0a9d WatchSource:0}: Error finding container 202091dc883856e6d9780dee48f162a37fcafa1775484696235b4ea2699b0a9d: Status 404 returned error can't find the container with id 202091dc883856e6d9780dee48f162a37fcafa1775484696235b4ea2699b0a9d
Apr 24 23:13:36.012215 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.012197    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b0a14a-b549-4724-8c64-7ede4ab92864-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:13:36.012270 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.012216    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrtl\" (UniqueName: \"kubernetes.io/projected/88b0a14a-b549-4724-8c64-7ede4ab92864-kube-api-access-qwrtl\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:13:36.012270 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.012230    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b0a14a-b549-4724-8c64-7ede4ab92864-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:13:36.012270 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.012240    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b0a14a-b549-4724-8c64-7ede4ab92864-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:13:36.026446 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.026421    2543 generic.go:358] "Generic (PLEG): container finished" podID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerID="86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59" exitCode=0
Apr 24 23:13:36.026553 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.026495    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"
Apr 24 23:13:36.026617 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.026499    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" event={"ID":"88b0a14a-b549-4724-8c64-7ede4ab92864","Type":"ContainerDied","Data":"86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59"}
Apr 24 23:13:36.026681 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.026626    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw" event={"ID":"88b0a14a-b549-4724-8c64-7ede4ab92864","Type":"ContainerDied","Data":"24b9dd7b8805a6aef55ca5491b3f4ae585c03283173f03f9d78639454e98b73b"}
Apr 24 23:13:36.026681 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.026656    2543 scope.go:117] "RemoveContainer" containerID="2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a"
Apr 24 23:13:36.027841 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.027825    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" event={"ID":"2ea16570-90f9-4f35-be85-00593496a913","Type":"ContainerStarted","Data":"202091dc883856e6d9780dee48f162a37fcafa1775484696235b4ea2699b0a9d"}
Apr 24 23:13:36.045995 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.045976    2543 scope.go:117] "RemoveContainer" containerID="86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59"
Apr 24 23:13:36.065144 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.065123    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"]
Apr 24 23:13:36.069759 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.069734    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-d64d94cdb-dmwhw"]
Apr 24 23:13:36.071832 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.071817    2543 scope.go:117] "RemoveContainer" containerID="48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7"
Apr 24 23:13:36.078760 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.078742    2543 scope.go:117] "RemoveContainer" containerID="2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a"
Apr 24 23:13:36.079004 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:13:36.078987    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a\": container with ID starting with 2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a not found: ID does not exist" containerID="2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a"
Apr 24 23:13:36.079056 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.079012    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a"} err="failed to get container status \"2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a\": rpc error: code = NotFound desc = could not find container \"2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a\": container with ID starting with 2f1e7570e42df597b23dbd4c9d9c869492a9ae2f9009762aae03ef20514a2f6a not found: ID does not exist"
Apr 24 23:13:36.079056 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.079029    2543 scope.go:117] "RemoveContainer" containerID="86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59"
Apr 24 23:13:36.079317 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:13:36.079295    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59\": container with ID starting with 86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59 not found: ID does not exist" containerID="86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59"
Apr 24 23:13:36.079393 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.079350    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59"} err="failed to get container status \"86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59\": rpc error: code = NotFound desc = could not find container \"86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59\": container with ID starting with 86bd27a5a4351651fca1ec397df5a040dea280e134411173336ea74286e60e59 not found: ID does not exist"
Apr 24 23:13:36.079393 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.079371    2543 scope.go:117] "RemoveContainer" containerID="48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7"
Apr 24 23:13:36.079712 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:13:36.079687    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7\": container with ID starting with 48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7 not found: ID does not exist" containerID="48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7"
Apr 24 23:13:36.079798 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:36.079715    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7"} err="failed to get container status \"48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7\": rpc error: code = NotFound desc = could not find container \"48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7\": container with ID starting with 48e0fc43717b20f940f76dcbae8bee491bb43018e4a6ed2f77752fa903ed18f7 not found: ID does not exist"
Apr 24 23:13:37.032218 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:37.032176    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" event={"ID":"2ea16570-90f9-4f35-be85-00593496a913","Type":"ContainerStarted","Data":"adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5"}
Apr 24 23:13:37.055279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:37.055246    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" path="/var/lib/kubelet/pods/88b0a14a-b549-4724-8c64-7ede4ab92864/volumes"
Apr 24 23:13:40.042362 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:40.042311    2543 generic.go:358] "Generic (PLEG): container finished" podID="2ea16570-90f9-4f35-be85-00593496a913" containerID="adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5" exitCode=0
Apr 24 23:13:40.042776 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:40.042379    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" event={"ID":"2ea16570-90f9-4f35-be85-00593496a913","Type":"ContainerDied","Data":"adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5"}
Apr 24 23:13:41.046809 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:41.046766    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" event={"ID":"2ea16570-90f9-4f35-be85-00593496a913","Type":"ContainerStarted","Data":"65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905"}
Apr 24 23:13:41.047153 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:41.046818    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" event={"ID":"2ea16570-90f9-4f35-be85-00593496a913","Type":"ContainerStarted","Data":"4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e"}
Apr 24 23:13:41.047153 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:41.047020    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:41.066327 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:41.066283    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" podStartSLOduration=7.066271088 podStartE2EDuration="7.066271088s" podCreationTimestamp="2026-04-24 23:13:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:13:41.06492313 +0000 UTC m=+2624.491320661" watchObservedRunningTime="2026-04-24 23:13:41.066271088 +0000 UTC m=+2624.492668618"
Apr 24 23:13:42.049849 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:42.049815    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:13:48.057660 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:13:48.057633    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:14:18.110124 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:18.110090    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Apr 24 23:14:28.060105 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:28.060079    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:14:35.062134 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.062106    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"]
Apr 24 23:14:35.062675 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.062419    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kserve-container" containerID="cri-o://4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e" gracePeriod=30
Apr 24 23:14:35.062675 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.062488    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kube-rbac-proxy" containerID="cri-o://65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905" gracePeriod=30
Apr 24 23:14:35.132454 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.132417    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"]
Apr 24 23:14:35.132738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.132726    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kube-rbac-proxy"
Apr 24 23:14:35.132798 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.132740    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kube-rbac-proxy"
Apr 24 23:14:35.132798 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.132748    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kserve-container"
Apr 24 23:14:35.132798 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.132753    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kserve-container"
Apr 24 23:14:35.132798 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.132761    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="storage-initializer"
Apr 24 23:14:35.132798 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.132767    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="storage-initializer"
Apr 24 23:14:35.132951 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.132820    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kube-rbac-proxy"
Apr 24 23:14:35.132951 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.132829    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="88b0a14a-b549-4724-8c64-7ede4ab92864" containerName="kserve-container"
Apr 24 23:14:35.135959 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.135944    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.138333 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.138285    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Apr 24 23:14:35.138514 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.138496    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Apr 24 23:14:35.145249 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.145227    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"]
Apr 24 23:14:35.198324 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.198298    2543 generic.go:358] "Generic (PLEG): container finished" podID="2ea16570-90f9-4f35-be85-00593496a913" containerID="65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905" exitCode=2
Apr 24 23:14:35.198424 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.198366    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" event={"ID":"2ea16570-90f9-4f35-be85-00593496a913","Type":"ContainerDied","Data":"65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905"}
Apr 24 23:14:35.245823 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.245800    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e34adb-e5d4-4588-bb36-d1351308e137-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.245895 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.245838    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e34adb-e5d4-4588-bb36-d1351308e137-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.245895 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.245874    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdmr\" (UniqueName: \"kubernetes.io/projected/16e34adb-e5d4-4588-bb36-d1351308e137-kube-api-access-2zdmr\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.246015 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.245915    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e34adb-e5d4-4588-bb36-d1351308e137-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.346770 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.346715    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e34adb-e5d4-4588-bb36-d1351308e137-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.346770 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.346749    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e34adb-e5d4-4588-bb36-d1351308e137-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.346899 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.346783    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e34adb-e5d4-4588-bb36-d1351308e137-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.346899 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.346820    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdmr\" (UniqueName: \"kubernetes.io/projected/16e34adb-e5d4-4588-bb36-d1351308e137-kube-api-access-2zdmr\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.346899 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:14:35.346855    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Apr 24 23:14:35.347019 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:14:35.346922    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16e34adb-e5d4-4588-bb36-d1351308e137-proxy-tls podName:16e34adb-e5d4-4588-bb36-d1351308e137 nodeName:}" failed. No retries permitted until 2026-04-24 23:14:35.846906718 +0000 UTC m=+2679.273304226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/16e34adb-e5d4-4588-bb36-d1351308e137-proxy-tls") pod "isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" (UID: "16e34adb-e5d4-4588-bb36-d1351308e137") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Apr 24 23:14:35.347126 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.347109    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e34adb-e5d4-4588-bb36-d1351308e137-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.347490 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.347471    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e34adb-e5d4-4588-bb36-d1351308e137-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.355777 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.355751    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdmr\" (UniqueName: \"kubernetes.io/projected/16e34adb-e5d4-4588-bb36-d1351308e137-kube-api-access-2zdmr\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.850595 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.850565    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e34adb-e5d4-4588-bb36-d1351308e137-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:35.853126 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:35.853091    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e34adb-e5d4-4588-bb36-d1351308e137-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:36.046485 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:36.046457    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:36.168528 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:36.168502    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"]
Apr 24 23:14:36.170847 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:14:36.170815    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e34adb_e5d4_4588_bb36_d1351308e137.slice/crio-65c0e0cb1f134444fd3bd394e42a0427d0f04e99256f247ef73932747e8d9325 WatchSource:0}: Error finding container 65c0e0cb1f134444fd3bd394e42a0427d0f04e99256f247ef73932747e8d9325: Status 404 returned error can't find the container with id 65c0e0cb1f134444fd3bd394e42a0427d0f04e99256f247ef73932747e8d9325
Apr 24 23:14:36.201954 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:36.201927    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" event={"ID":"16e34adb-e5d4-4588-bb36-d1351308e137","Type":"ContainerStarted","Data":"65c0e0cb1f134444fd3bd394e42a0427d0f04e99256f247ef73932747e8d9325"}
Apr 24 23:14:37.205938 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:37.205857    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" event={"ID":"16e34adb-e5d4-4588-bb36-d1351308e137","Type":"ContainerStarted","Data":"9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e"}
Apr 24 23:14:38.052786 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:38.052743    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Apr 24 23:14:40.215078 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:40.215051    2543 generic.go:358] "Generic (PLEG): container finished" podID="16e34adb-e5d4-4588-bb36-d1351308e137" containerID="9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e" exitCode=0
Apr 24 23:14:40.215464 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:40.215088    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" event={"ID":"16e34adb-e5d4-4588-bb36-d1351308e137","Type":"ContainerDied","Data":"9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e"}
Apr 24 23:14:41.220152 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.220118    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" event={"ID":"16e34adb-e5d4-4588-bb36-d1351308e137","Type":"ContainerStarted","Data":"831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c"}
Apr 24 23:14:41.220152 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.220156    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" event={"ID":"16e34adb-e5d4-4588-bb36-d1351308e137","Type":"ContainerStarted","Data":"bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df"}
Apr 24 23:14:41.220592 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.220452    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:41.220592 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.220549    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:41.221777 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.221752    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Apr 24 23:14:41.241104 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.241057    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podStartSLOduration=6.241046556 podStartE2EDuration="6.241046556s" podCreationTimestamp="2026-04-24 23:14:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:14:41.238447963 +0000 UTC m=+2684.664845503" watchObservedRunningTime="2026-04-24 23:14:41.241046556 +0000 UTC m=+2684.667444086"
Apr 24 23:14:41.497072 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.497051    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:14:41.593835 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.593805    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea16570-90f9-4f35-be85-00593496a913-proxy-tls\") pod \"2ea16570-90f9-4f35-be85-00593496a913\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") "
Apr 24 23:14:41.594003 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.593842    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea16570-90f9-4f35-be85-00593496a913-kserve-provision-location\") pod \"2ea16570-90f9-4f35-be85-00593496a913\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") "
Apr 24 23:14:41.594003 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.593898    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzw9q\" (UniqueName: \"kubernetes.io/projected/2ea16570-90f9-4f35-be85-00593496a913-kube-api-access-jzw9q\") pod \"2ea16570-90f9-4f35-be85-00593496a913\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") "
Apr 24 23:14:41.594003 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.593928    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea16570-90f9-4f35-be85-00593496a913-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"2ea16570-90f9-4f35-be85-00593496a913\" (UID: \"2ea16570-90f9-4f35-be85-00593496a913\") "
Apr 24 23:14:41.594240 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.594217    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea16570-90f9-4f35-be85-00593496a913-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ea16570-90f9-4f35-be85-00593496a913" (UID: "2ea16570-90f9-4f35-be85-00593496a913"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:14:41.594317 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.594297    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea16570-90f9-4f35-be85-00593496a913-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "2ea16570-90f9-4f35-be85-00593496a913" (UID: "2ea16570-90f9-4f35-be85-00593496a913"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:14:41.595953 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.595934    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea16570-90f9-4f35-be85-00593496a913-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ea16570-90f9-4f35-be85-00593496a913" (UID: "2ea16570-90f9-4f35-be85-00593496a913"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:14:41.596069 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.596050    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea16570-90f9-4f35-be85-00593496a913-kube-api-access-jzw9q" (OuterVolumeSpecName: "kube-api-access-jzw9q") pod "2ea16570-90f9-4f35-be85-00593496a913" (UID: "2ea16570-90f9-4f35-be85-00593496a913"). InnerVolumeSpecName "kube-api-access-jzw9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:14:41.694887 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.694858    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzw9q\" (UniqueName: \"kubernetes.io/projected/2ea16570-90f9-4f35-be85-00593496a913-kube-api-access-jzw9q\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:14:41.694887 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.694885    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ea16570-90f9-4f35-be85-00593496a913-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:14:41.695060 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.694895    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ea16570-90f9-4f35-be85-00593496a913-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:14:41.695060 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:41.694905    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ea16570-90f9-4f35-be85-00593496a913-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:14:42.224988 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.224953    2543 generic.go:358] "Generic (PLEG): container finished" podID="2ea16570-90f9-4f35-be85-00593496a913" containerID="4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e" exitCode=0
Apr 24 23:14:42.225476 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.225048    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"
Apr 24 23:14:42.225476 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.225048    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" event={"ID":"2ea16570-90f9-4f35-be85-00593496a913","Type":"ContainerDied","Data":"4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e"}
Apr 24 23:14:42.225476 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.225089    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25" event={"ID":"2ea16570-90f9-4f35-be85-00593496a913","Type":"ContainerDied","Data":"202091dc883856e6d9780dee48f162a37fcafa1775484696235b4ea2699b0a9d"}
Apr 24 23:14:42.225476 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.225105    2543 scope.go:117] "RemoveContainer" containerID="65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905"
Apr 24 23:14:42.225866 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.225841    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Apr 24 23:14:42.232970 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.232951    2543 scope.go:117] "RemoveContainer" containerID="4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e"
Apr 24 23:14:42.245301 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.245283    2543 scope.go:117] "RemoveContainer" containerID="adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5"
Apr 24 23:14:42.246273 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.246092    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"]
Apr 24 23:14:42.247903 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.247886    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-swr25"]
Apr 24 23:14:42.252470 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.252452    2543 scope.go:117] "RemoveContainer" containerID="65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905"
Apr 24 23:14:42.252714 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:14:42.252697    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905\": container with ID starting with 65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905 not found: ID does not exist" containerID="65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905"
Apr 24 23:14:42.252765 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.252720    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905"} err="failed to get container status \"65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905\": rpc error: code = NotFound desc = could not find container \"65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905\": container with ID starting with 65c56f0105e00891ff9b80930e5e8d47d9ed467bcf57a072bb8a9ad982380905 not found: ID does not exist"
Apr 24 23:14:42.252765 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.252738    2543 scope.go:117] "RemoveContainer" containerID="4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e"
Apr 24 23:14:42.252974 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:14:42.252959    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e\": container with ID starting with 4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e not found: ID does not exist" containerID="4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e"
Apr 24 23:14:42.253025 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.252976    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e"} err="failed to get container status \"4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e\": rpc error: code = NotFound desc = could not find container \"4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e\": container with ID starting with 4ac1c6135aefd609a48367d82b6cac504e0f8fa8054bb222a8ac263c85e1017e not found: ID does not exist"
Apr 24 23:14:42.253025 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.252989    2543 scope.go:117] "RemoveContainer" containerID="adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5"
Apr 24 23:14:42.253166 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:14:42.253152    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5\": container with ID starting with adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5 not found: ID does not exist" containerID="adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5"
Apr 24 23:14:42.253200 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:42.253170    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5"} err="failed to get container status \"adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5\": rpc error: code = NotFound desc = could not find container \"adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5\": container with ID starting with adfeb7334c6d55238c9f7f7a6dd38722fcf57d25b470e40025dca42cb4a8ffe5 not found: ID does not exist"
Apr 24 23:14:43.055619 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:43.055583    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea16570-90f9-4f35-be85-00593496a913" path="/var/lib/kubelet/pods/2ea16570-90f9-4f35-be85-00593496a913/volumes"
Apr 24 23:14:47.229506 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:47.229480    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:14:47.229973 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:47.229949    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Apr 24 23:14:57.230273 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:57.230237    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Apr 24 23:14:57.552365 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:57.552270    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:14:57.557475 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:14:57.557457    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:15:07.230529 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:07.230489    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Apr 24 23:15:17.230220 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:17.230174    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Apr 24 23:15:27.230704 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:27.230668    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Apr 24 23:15:37.230860 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:37.230823    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Apr 24 23:15:47.230519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:47.230492    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:15:55.379530 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.379495    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"]
Apr 24 23:15:55.379932 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.379803    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" containerID="cri-o://bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df" gracePeriod=30
Apr 24 23:15:55.379932 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.379861    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kube-rbac-proxy" containerID="cri-o://831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c" gracePeriod=30
Apr 24 23:15:55.397599 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.397576    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"]
Apr 24 23:15:55.397874 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.397863    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="storage-initializer"
Apr 24 23:15:55.397918 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.397877    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="storage-initializer"
Apr 24 23:15:55.397918 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.397891    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kube-rbac-proxy"
Apr 24 23:15:55.397918 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.397896    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kube-rbac-proxy"
Apr 24 23:15:55.397918 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.397909    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kserve-container"
Apr 24 23:15:55.397918 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.397914    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kserve-container"
Apr 24 23:15:55.398068 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.397955    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kube-rbac-proxy"
Apr 24 23:15:55.398068 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.397966    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="2ea16570-90f9-4f35-be85-00593496a913" containerName="kserve-container"
Apr 24 23:15:55.400896 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.400881    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.403019 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.402999    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Apr 24 23:15:55.403090 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.403018    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Apr 24 23:15:55.409220 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.409202    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"]
Apr 24 23:15:55.422010 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.421989    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.422107 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.422026    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.422107 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.422064    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtxc2\" (UniqueName: \"kubernetes.io/projected/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kube-api-access-wtxc2\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.422187 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.422111    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.523072 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.523049    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.523153 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.523087    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.523153 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.523115    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.523228 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.523163    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wtxc2\" (UniqueName: \"kubernetes.io/projected/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kube-api-access-wtxc2\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.523304 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:15:55.523287    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Apr 24 23:15:55.523404 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:15:55.523392    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-proxy-tls podName:bf68f84e-bc39-43cf-9a5b-19a8277cbf38 nodeName:}" failed. No retries permitted until 2026-04-24 23:15:56.023372732 +0000 UTC m=+2759.449770248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" (UID: "bf68f84e-bc39-43cf-9a5b-19a8277cbf38") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Apr 24 23:15:55.523464 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.523410    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.523719 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.523700    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:55.533707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:55.533678    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtxc2\" (UniqueName: \"kubernetes.io/projected/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kube-api-access-wtxc2\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:56.026898 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:56.026862    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:56.032696 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:56.030417    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:56.310402 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:56.310307    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:15:56.430491 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:56.430449    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"]
Apr 24 23:15:56.433503 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:15:56.433469    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf68f84e_bc39_43cf_9a5b_19a8277cbf38.slice/crio-743efaccef3824ddbc6674965b4f591397724f4cc66a6596b7d76f24dffdd41a WatchSource:0}: Error finding container 743efaccef3824ddbc6674965b4f591397724f4cc66a6596b7d76f24dffdd41a: Status 404 returned error can't find the container with id 743efaccef3824ddbc6674965b4f591397724f4cc66a6596b7d76f24dffdd41a
Apr 24 23:15:56.435288 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:56.435269    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 23:15:56.440681 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:56.440655    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" event={"ID":"bf68f84e-bc39-43cf-9a5b-19a8277cbf38","Type":"ContainerStarted","Data":"743efaccef3824ddbc6674965b4f591397724f4cc66a6596b7d76f24dffdd41a"}
Apr 24 23:15:56.442551 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:56.442527    2543 generic.go:358] "Generic (PLEG): container finished" podID="16e34adb-e5d4-4588-bb36-d1351308e137" containerID="831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c" exitCode=2
Apr 24 23:15:56.442651 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:56.442567    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" event={"ID":"16e34adb-e5d4-4588-bb36-d1351308e137","Type":"ContainerDied","Data":"831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c"}
Apr 24 23:15:57.225821 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:57.225780    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Apr 24 23:15:57.229991 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:57.229969    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Apr 24 23:15:57.446972 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:57.446937    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" event={"ID":"bf68f84e-bc39-43cf-9a5b-19a8277cbf38","Type":"ContainerStarted","Data":"b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453"}
Apr 24 23:15:59.423465 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.423442    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:15:59.451053 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.451025    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e34adb-e5d4-4588-bb36-d1351308e137-proxy-tls\") pod \"16e34adb-e5d4-4588-bb36-d1351308e137\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") "
Apr 24 23:15:59.451195 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.451084    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zdmr\" (UniqueName: \"kubernetes.io/projected/16e34adb-e5d4-4588-bb36-d1351308e137-kube-api-access-2zdmr\") pod \"16e34adb-e5d4-4588-bb36-d1351308e137\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") "
Apr 24 23:15:59.451263 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.451186    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e34adb-e5d4-4588-bb36-d1351308e137-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"16e34adb-e5d4-4588-bb36-d1351308e137\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") "
Apr 24 23:15:59.451263 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.451231    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e34adb-e5d4-4588-bb36-d1351308e137-kserve-provision-location\") pod \"16e34adb-e5d4-4588-bb36-d1351308e137\" (UID: \"16e34adb-e5d4-4588-bb36-d1351308e137\") "
Apr 24 23:15:59.451611 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.451572    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e34adb-e5d4-4588-bb36-d1351308e137-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "16e34adb-e5d4-4588-bb36-d1351308e137" (UID: "16e34adb-e5d4-4588-bb36-d1351308e137"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:15:59.451830 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.451808    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e34adb-e5d4-4588-bb36-d1351308e137-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16e34adb-e5d4-4588-bb36-d1351308e137" (UID: "16e34adb-e5d4-4588-bb36-d1351308e137"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:15:59.453609 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.453584    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e34adb-e5d4-4588-bb36-d1351308e137-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16e34adb-e5d4-4588-bb36-d1351308e137" (UID: "16e34adb-e5d4-4588-bb36-d1351308e137"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:15:59.453753 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.453727    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e34adb-e5d4-4588-bb36-d1351308e137-kube-api-access-2zdmr" (OuterVolumeSpecName: "kube-api-access-2zdmr") pod "16e34adb-e5d4-4588-bb36-d1351308e137" (UID: "16e34adb-e5d4-4588-bb36-d1351308e137"). InnerVolumeSpecName "kube-api-access-2zdmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:15:59.458428 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.458375    2543 generic.go:358] "Generic (PLEG): container finished" podID="16e34adb-e5d4-4588-bb36-d1351308e137" containerID="bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df" exitCode=0
Apr 24 23:15:59.458428 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.458408    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" event={"ID":"16e34adb-e5d4-4588-bb36-d1351308e137","Type":"ContainerDied","Data":"bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df"}
Apr 24 23:15:59.458588 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.458453    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb" event={"ID":"16e34adb-e5d4-4588-bb36-d1351308e137","Type":"ContainerDied","Data":"65c0e0cb1f134444fd3bd394e42a0427d0f04e99256f247ef73932747e8d9325"}
Apr 24 23:15:59.458588 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.458475    2543 scope.go:117] "RemoveContainer" containerID="831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c"
Apr 24 23:15:59.458588 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.458503    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"
Apr 24 23:15:59.471174 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.471159    2543 scope.go:117] "RemoveContainer" containerID="bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df"
Apr 24 23:15:59.477941 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.477924    2543 scope.go:117] "RemoveContainer" containerID="9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e"
Apr 24 23:15:59.482640 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.482621    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"]
Apr 24 23:15:59.485245 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.485224    2543 scope.go:117] "RemoveContainer" containerID="831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c"
Apr 24 23:15:59.485564 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:15:59.485546    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c\": container with ID starting with 831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c not found: ID does not exist" containerID="831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c"
Apr 24 23:15:59.485639 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.485571    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c"} err="failed to get container status \"831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c\": rpc error: code = NotFound desc = could not find container \"831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c\": container with ID starting with 831285b723e77550f39c9b9393b74f3838d6511d8aae5c74f2a2005d7c30829c not found: ID does not exist"
Apr 24 23:15:59.485639 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.485588    2543 scope.go:117] "RemoveContainer" containerID="bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df"
Apr 24 23:15:59.485870 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:15:59.485851    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df\": container with ID starting with bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df not found: ID does not exist" containerID="bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df"
Apr 24 23:15:59.485909 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.485881    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df"} err="failed to get container status \"bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df\": rpc error: code = NotFound desc = could not find container \"bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df\": container with ID starting with bb7365cc7f5d70cf903550b543e07c6c572efcc5bff2dcb3ac8d6716406dd8df not found: ID does not exist"
Apr 24 23:15:59.485909 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.485899    2543 scope.go:117] "RemoveContainer" containerID="9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e"
Apr 24 23:15:59.486147 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:15:59.486130    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e\": container with ID starting with 9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e not found: ID does not exist" containerID="9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e"
Apr 24 23:15:59.486191 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.486155    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7c66c8c59d-6k9jb"]
Apr 24 23:15:59.486191 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.486152    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e"} err="failed to get container status \"9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e\": rpc error: code = NotFound desc = could not find container \"9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e\": container with ID starting with 9624926aa060f66637e98a57700eb78bedaa0b6c2b6b8476f1dd616c0584c27e not found: ID does not exist"
Apr 24 23:15:59.552681 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.552651    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16e34adb-e5d4-4588-bb36-d1351308e137-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:15:59.552681 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.552679    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zdmr\" (UniqueName: \"kubernetes.io/projected/16e34adb-e5d4-4588-bb36-d1351308e137-kube-api-access-2zdmr\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:15:59.552819 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.552689    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16e34adb-e5d4-4588-bb36-d1351308e137-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:15:59.552819 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:15:59.552698    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16e34adb-e5d4-4588-bb36-d1351308e137-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:16:00.463296 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:00.463265    2543 generic.go:358] "Generic (PLEG): container finished" podID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerID="b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453" exitCode=0
Apr 24 23:16:00.463778 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:00.463360    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" event={"ID":"bf68f84e-bc39-43cf-9a5b-19a8277cbf38","Type":"ContainerDied","Data":"b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453"}
Apr 24 23:16:01.055542 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:01.055509    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" path="/var/lib/kubelet/pods/16e34adb-e5d4-4588-bb36-d1351308e137/volumes"
Apr 24 23:16:01.469303 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:01.469271    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" event={"ID":"bf68f84e-bc39-43cf-9a5b-19a8277cbf38","Type":"ContainerStarted","Data":"198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc"}
Apr 24 23:16:01.469752 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:01.469310    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" event={"ID":"bf68f84e-bc39-43cf-9a5b-19a8277cbf38","Type":"ContainerStarted","Data":"d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5"}
Apr 24 23:16:01.469752 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:01.469605    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:16:01.469837 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:01.469754    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:16:01.470899 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:01.470875    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 24 23:16:01.487619 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:01.487579    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podStartSLOduration=6.48756572 podStartE2EDuration="6.48756572s" podCreationTimestamp="2026-04-24 23:15:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:16:01.48608471 +0000 UTC m=+2764.912482240" watchObservedRunningTime="2026-04-24 23:16:01.48756572 +0000 UTC m=+2764.913963250"
Apr 24 23:16:02.471968 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:02.471929    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 24 23:16:07.475865 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:07.475790    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:16:07.476311 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:07.476286    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 24 23:16:17.476355 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:17.476292    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 24 23:16:27.476480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:27.476443    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 24 23:16:37.477055 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:37.477010    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 24 23:16:47.477127 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:47.477087    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 24 23:16:57.476927 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:16:57.476885    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 24 23:17:07.477499 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:07.477462    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:17:15.509737 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.509705    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"]
Apr 24 23:17:15.510237 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.510095    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" containerID="cri-o://d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5" gracePeriod=30
Apr 24 23:17:15.510237 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.510164    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kube-rbac-proxy" containerID="cri-o://198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc" gracePeriod=30
Apr 24 23:17:15.573721 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.573686    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"]
Apr 24 23:17:15.574045 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.574028    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="storage-initializer"
Apr 24 23:17:15.574128 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.574048    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="storage-initializer"
Apr 24 23:17:15.574128 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.574062    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kube-rbac-proxy"
Apr 24 23:17:15.574128 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.574070    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kube-rbac-proxy"
Apr 24 23:17:15.574128 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.574085    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container"
Apr 24 23:17:15.574128 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.574094    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container"
Apr 24 23:17:15.574438 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.574165    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kserve-container"
Apr 24 23:17:15.574438 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.574181    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="16e34adb-e5d4-4588-bb36-d1351308e137" containerName="kube-rbac-proxy"
Apr 24 23:17:15.578590 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.578571    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.580985 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.580968    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Apr 24 23:17:15.581077 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.580985    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Apr 24 23:17:15.584855 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.584834    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"]
Apr 24 23:17:15.620043 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.620019    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc8bw\" (UniqueName: \"kubernetes.io/projected/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kube-api-access-vc8bw\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.620137 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.620052    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.620137 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.620086    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.620215 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.620147    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.682058 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.682034    2543 generic.go:358] "Generic (PLEG): container finished" podID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerID="198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc" exitCode=2
Apr 24 23:17:15.682158 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.682106    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" event={"ID":"bf68f84e-bc39-43cf-9a5b-19a8277cbf38","Type":"ContainerDied","Data":"198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc"}
Apr 24 23:17:15.720651 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.720623    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.720764 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.720668    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.720764 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.720709    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.720764 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.720745    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vc8bw\" (UniqueName: \"kubernetes.io/projected/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kube-api-access-vc8bw\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.721020 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.720998    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.721461 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.721438    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.723254 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.723232    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.727857 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.727835    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc8bw\" (UniqueName: \"kubernetes.io/projected/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kube-api-access-vc8bw\") pod \"isvc-tensorflow-predictor-6756f669d7-fqjht\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:15.889124 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:15.889043    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:16.008900 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:16.008867    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"]
Apr 24 23:17:16.012512 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:17:16.012476    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90fc98c_c2a8_4d8d_8c32_6e9a17f08370.slice/crio-4c99be92f0c7b2446e5bb5c42dcdd533d69b9f4c3369a38f04eb3c40de9d3fe4 WatchSource:0}: Error finding container 4c99be92f0c7b2446e5bb5c42dcdd533d69b9f4c3369a38f04eb3c40de9d3fe4: Status 404 returned error can't find the container with id 4c99be92f0c7b2446e5bb5c42dcdd533d69b9f4c3369a38f04eb3c40de9d3fe4
Apr 24 23:17:16.686327 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:16.686291    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" event={"ID":"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370","Type":"ContainerStarted","Data":"85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8"}
Apr 24 23:17:16.686698 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:16.686351    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" event={"ID":"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370","Type":"ContainerStarted","Data":"4c99be92f0c7b2446e5bb5c42dcdd533d69b9f4c3369a38f04eb3c40de9d3fe4"}
Apr 24 23:17:17.472090 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:17.472051    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Apr 24 23:17:17.476381 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:17.476330    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Apr 24 23:17:19.258275 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.258253    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:17:19.348911 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.348890    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-proxy-tls\") pod \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") "
Apr 24 23:17:19.349034 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.348963    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtxc2\" (UniqueName: \"kubernetes.io/projected/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kube-api-access-wtxc2\") pod \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") "
Apr 24 23:17:19.349034 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.348994    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") "
Apr 24 23:17:19.349034 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.349016    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kserve-provision-location\") pod \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\" (UID: \"bf68f84e-bc39-43cf-9a5b-19a8277cbf38\") "
Apr 24 23:17:19.349372 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.349327    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "bf68f84e-bc39-43cf-9a5b-19a8277cbf38" (UID: "bf68f84e-bc39-43cf-9a5b-19a8277cbf38"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:17:19.349459 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.349389    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf68f84e-bc39-43cf-9a5b-19a8277cbf38" (UID: "bf68f84e-bc39-43cf-9a5b-19a8277cbf38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:17:19.351034 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.351011    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf68f84e-bc39-43cf-9a5b-19a8277cbf38" (UID: "bf68f84e-bc39-43cf-9a5b-19a8277cbf38"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:17:19.351093 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.351074    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kube-api-access-wtxc2" (OuterVolumeSpecName: "kube-api-access-wtxc2") pod "bf68f84e-bc39-43cf-9a5b-19a8277cbf38" (UID: "bf68f84e-bc39-43cf-9a5b-19a8277cbf38"). InnerVolumeSpecName "kube-api-access-wtxc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:17:19.449484 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.449438    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtxc2\" (UniqueName: \"kubernetes.io/projected/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kube-api-access-wtxc2\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:17:19.449484 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.449459    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:17:19.449484 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.449471    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:17:19.449484 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.449480    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf68f84e-bc39-43cf-9a5b-19a8277cbf38-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:17:19.697572 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.697539    2543 generic.go:358] "Generic (PLEG): container finished" podID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerID="d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5" exitCode=0
Apr 24 23:17:19.697707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.697616    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" event={"ID":"bf68f84e-bc39-43cf-9a5b-19a8277cbf38","Type":"ContainerDied","Data":"d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5"}
Apr 24 23:17:19.697707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.697639    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl" event={"ID":"bf68f84e-bc39-43cf-9a5b-19a8277cbf38","Type":"ContainerDied","Data":"743efaccef3824ddbc6674965b4f591397724f4cc66a6596b7d76f24dffdd41a"}
Apr 24 23:17:19.697707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.697667    2543 scope.go:117] "RemoveContainer" containerID="198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc"
Apr 24 23:17:19.697707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.697671    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"
Apr 24 23:17:19.705717 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.705701    2543 scope.go:117] "RemoveContainer" containerID="d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5"
Apr 24 23:17:19.712839 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.712822    2543 scope.go:117] "RemoveContainer" containerID="b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453"
Apr 24 23:17:19.718566 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.718542    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"]
Apr 24 23:17:19.720209 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.720193    2543 scope.go:117] "RemoveContainer" containerID="198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc"
Apr 24 23:17:19.720460 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:17:19.720442    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc\": container with ID starting with 198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc not found: ID does not exist" containerID="198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc"
Apr 24 23:17:19.720507 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.720471    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc"} err="failed to get container status \"198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc\": rpc error: code = NotFound desc = could not find container \"198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc\": container with ID starting with 198f6c00d02bc39b3b9e699bc502ba7e7309161740634b368528c82d3b62eddc not found: ID does not exist"
Apr 24 23:17:19.720507 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.720488    2543 scope.go:117] "RemoveContainer" containerID="d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5"
Apr 24 23:17:19.720721 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:17:19.720700    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5\": container with ID starting with d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5 not found: ID does not exist" containerID="d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5"
Apr 24 23:17:19.720787 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.720727    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5"} err="failed to get container status \"d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5\": rpc error: code = NotFound desc = could not find container \"d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5\": container with ID starting with d1eca81f193a714b5458826133ac342bf98b84b95c7de01fc8e4c6aa614103e5 not found: ID does not exist"
Apr 24 23:17:19.720787 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.720746    2543 scope.go:117] "RemoveContainer" containerID="b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453"
Apr 24 23:17:19.721133 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:17:19.721026    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453\": container with ID starting with b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453 not found: ID does not exist" containerID="b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453"
Apr 24 23:17:19.721208 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.721137    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453"} err="failed to get container status \"b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453\": rpc error: code = NotFound desc = could not find container \"b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453\": container with ID starting with b7f7921539be2fe0005659071f9d587736331dc9fbb5c22ae77f9a1bb9990453 not found: ID does not exist"
Apr 24 23:17:19.723277 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:19.723256    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-748989b567-ggmcl"]
Apr 24 23:17:20.701643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:20.701616    2543 generic.go:358] "Generic (PLEG): container finished" podID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerID="85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8" exitCode=0
Apr 24 23:17:20.701979 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:20.701689    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" event={"ID":"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370","Type":"ContainerDied","Data":"85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8"}
Apr 24 23:17:21.055735 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:21.055705    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" path="/var/lib/kubelet/pods/bf68f84e-bc39-43cf-9a5b-19a8277cbf38/volumes"
Apr 24 23:17:24.720397 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:24.720365    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" event={"ID":"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370","Type":"ContainerStarted","Data":"1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c"}
Apr 24 23:17:24.720711 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:24.720408    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" event={"ID":"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370","Type":"ContainerStarted","Data":"f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02"}
Apr 24 23:17:24.720711 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:24.720620    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:24.739628 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:24.739583    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podStartSLOduration=5.887690196 podStartE2EDuration="9.739570781s" podCreationTimestamp="2026-04-24 23:17:15 +0000 UTC" firstStartedPulling="2026-04-24 23:17:20.702935841 +0000 UTC m=+2844.129333350" lastFinishedPulling="2026-04-24 23:17:24.554816428 +0000 UTC m=+2847.981213935" observedRunningTime="2026-04-24 23:17:24.738758791 +0000 UTC m=+2848.165156323" watchObservedRunningTime="2026-04-24 23:17:24.739570781 +0000 UTC m=+2848.165968310"
Apr 24 23:17:25.723593 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:25.723561    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:25.724710 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:25.724688    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 24 23:17:26.726996 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:26.726956    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 24 23:17:31.731322 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:31.731293    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:31.731845 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:31.731816    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Apr 24 23:17:41.733019 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:41.732946    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:17:56.050162 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.050128    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"]
Apr 24 23:17:56.050638 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.050503    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kserve-container" containerID="cri-o://f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02" gracePeriod=30
Apr 24 23:17:56.050638 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.050579    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy" containerID="cri-o://1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c" gracePeriod=30
Apr 24 23:17:56.137495 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.137463    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"]
Apr 24 23:17:56.137770 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.137758    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kube-rbac-proxy"
Apr 24 23:17:56.137813 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.137772    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kube-rbac-proxy"
Apr 24 23:17:56.137813 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.137783    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="storage-initializer"
Apr 24 23:17:56.137813 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.137789    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="storage-initializer"
Apr 24 23:17:56.137813 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.137804    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container"
Apr 24 23:17:56.137813 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.137810    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container"
Apr 24 23:17:56.137971 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.137856    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kube-rbac-proxy"
Apr 24 23:17:56.137971 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.137867    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="bf68f84e-bc39-43cf-9a5b-19a8277cbf38" containerName="kserve-container"
Apr 24 23:17:56.140825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.140811    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.143232 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.143211    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Apr 24 23:17:56.143410 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.143214    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Apr 24 23:17:56.151360 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.151324    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"]
Apr 24 23:17:56.221138 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.221109    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzdh\" (UniqueName: \"kubernetes.io/projected/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kube-api-access-gbzdh\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.221234 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.221170    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.221234 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.221222    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.221370 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.221249    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.321600 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.321545    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzdh\" (UniqueName: \"kubernetes.io/projected/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kube-api-access-gbzdh\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.321600 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.321589    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.321726 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.321637    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.321726 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.321662    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.321987 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.321970    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.322309 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.322290    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.324054 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.324032    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.329387 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.329367    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzdh\" (UniqueName: \"kubernetes.io/projected/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kube-api-access-gbzdh\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-vn98w\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.450823 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.450793    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:17:56.566738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.566717    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"]
Apr 24 23:17:56.569162 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:17:56.569130    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3aa846e_1fdf_4cae_8ecc_8916ef89c8bb.slice/crio-b882bba716cb1be13be9f3bd2ecfd13b6d9b2f3b0c0316d3073c7afc90b089d3 WatchSource:0}: Error finding container b882bba716cb1be13be9f3bd2ecfd13b6d9b2f3b0c0316d3073c7afc90b089d3: Status 404 returned error can't find the container with id b882bba716cb1be13be9f3bd2ecfd13b6d9b2f3b0c0316d3073c7afc90b089d3
Apr 24 23:17:56.727751 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.727716    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Apr 24 23:17:56.816031 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.815991    2543 generic.go:358] "Generic (PLEG): container finished" podID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerID="1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c" exitCode=2
Apr 24 23:17:56.816171 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.816076    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" event={"ID":"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370","Type":"ContainerDied","Data":"1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c"}
Apr 24 23:17:56.817469 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.817443    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" event={"ID":"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb","Type":"ContainerStarted","Data":"d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd"}
Apr 24 23:17:56.817569 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:17:56.817478    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" event={"ID":"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb","Type":"ContainerStarted","Data":"b882bba716cb1be13be9f3bd2ecfd13b6d9b2f3b0c0316d3073c7afc90b089d3"}
Apr 24 23:18:01.728266 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:01.728215    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Apr 24 23:18:01.834229 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:01.834199    2543 generic.go:358] "Generic (PLEG): container finished" podID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerID="d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd" exitCode=0
Apr 24 23:18:01.834431 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:01.834275    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" event={"ID":"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb","Type":"ContainerDied","Data":"d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd"}
Apr 24 23:18:02.838186 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:02.838158    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" event={"ID":"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb","Type":"ContainerStarted","Data":"50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5"}
Apr 24 23:18:02.838186 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:02.838191    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" event={"ID":"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb","Type":"ContainerStarted","Data":"96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098"}
Apr 24 23:18:02.838589 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:02.838385    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:18:02.855543 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:02.855498    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podStartSLOduration=6.855483151 podStartE2EDuration="6.855483151s" podCreationTimestamp="2026-04-24 23:17:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:18:02.854821308 +0000 UTC m=+2886.281218838" watchObservedRunningTime="2026-04-24 23:18:02.855483151 +0000 UTC m=+2886.281880683"
Apr 24 23:18:03.842370 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:03.842324    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:18:03.843521 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:03.843491    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Apr 24 23:18:04.845157 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:04.845120    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Apr 24 23:18:06.728026 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:06.727982    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Apr 24 23:18:06.728418 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:06.728091    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:18:09.850072 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:09.850044    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:18:09.850656 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:09.850626    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Apr 24 23:18:11.727927 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:11.727891    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Apr 24 23:18:16.732961 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:16.732907    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Apr 24 23:18:19.850978 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:19.850950    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:18:21.728121 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:21.728078    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Apr 24 23:18:26.686732 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.686704    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:18:26.849994 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.849912    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-proxy-tls\") pod \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") "
Apr 24 23:18:26.849994 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.849976    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") "
Apr 24 23:18:26.850209 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.850011    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kserve-provision-location\") pod \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") "
Apr 24 23:18:26.850209 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.850168    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc8bw\" (UniqueName: \"kubernetes.io/projected/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kube-api-access-vc8bw\") pod \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\" (UID: \"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370\") "
Apr 24 23:18:26.850407 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.850376    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" (UID: "e90fc98c-c2a8-4d8d-8c32-6e9a17f08370"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:18:26.852135 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.852115    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" (UID: "e90fc98c-c2a8-4d8d-8c32-6e9a17f08370"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:18:26.852304 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.852279    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kube-api-access-vc8bw" (OuterVolumeSpecName: "kube-api-access-vc8bw") pod "e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" (UID: "e90fc98c-c2a8-4d8d-8c32-6e9a17f08370"). InnerVolumeSpecName "kube-api-access-vc8bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:18:26.860784 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.860760    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" (UID: "e90fc98c-c2a8-4d8d-8c32-6e9a17f08370"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:18:26.906586 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.906556    2543 generic.go:358] "Generic (PLEG): container finished" podID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerID="f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02" exitCode=137
Apr 24 23:18:26.906662 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.906645    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"
Apr 24 23:18:26.906728 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.906642    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" event={"ID":"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370","Type":"ContainerDied","Data":"f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02"}
Apr 24 23:18:26.906770 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.906743    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht" event={"ID":"e90fc98c-c2a8-4d8d-8c32-6e9a17f08370","Type":"ContainerDied","Data":"4c99be92f0c7b2446e5bb5c42dcdd533d69b9f4c3369a38f04eb3c40de9d3fe4"}
Apr 24 23:18:26.906770 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.906760    2543 scope.go:117] "RemoveContainer" containerID="1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c"
Apr 24 23:18:26.914913 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.914699    2543 scope.go:117] "RemoveContainer" containerID="f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02"
Apr 24 23:18:26.922111 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.922094    2543 scope.go:117] "RemoveContainer" containerID="85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8"
Apr 24 23:18:26.927808 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.927786    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"]
Apr 24 23:18:26.928780 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.928765    2543 scope.go:117] "RemoveContainer" containerID="1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c"
Apr 24 23:18:26.929080 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:18:26.929052    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c\": container with ID starting with 1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c not found: ID does not exist" containerID="1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c"
Apr 24 23:18:26.929173 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.929097    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c"} err="failed to get container status \"1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c\": rpc error: code = NotFound desc = could not find container \"1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c\": container with ID starting with 1c725f4d002ae25f617d88d79e096cb063063a0b87af0466d66472b8156b703c not found: ID does not exist"
Apr 24 23:18:26.929173 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.929120    2543 scope.go:117] "RemoveContainer" containerID="f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02"
Apr 24 23:18:26.929581 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:18:26.929554    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02\": container with ID starting with f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02 not found: ID does not exist" containerID="f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02"
Apr 24 23:18:26.929668 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.929590    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02"} err="failed to get container status \"f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02\": rpc error: code = NotFound desc = could not find container \"f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02\": container with ID starting with f0e2f2b00889db15cbc291e6a59bcb8ba577d0806d235c811dd6a13ade603b02 not found: ID does not exist"
Apr 24 23:18:26.929668 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.929612    2543 scope.go:117] "RemoveContainer" containerID="85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8"
Apr 24 23:18:26.929915 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:18:26.929897    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8\": container with ID starting with 85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8 not found: ID does not exist" containerID="85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8"
Apr 24 23:18:26.929990 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.929919    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8"} err="failed to get container status \"85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8\": rpc error: code = NotFound desc = could not find container \"85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8\": container with ID starting with 85203fb02e2bd2ce5396cc5ea6849ff24e56d6342e33d03040eea782e12feea8 not found: ID does not exist"
Apr 24 23:18:26.930993 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.930974    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-fqjht"]
Apr 24 23:18:26.950722 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.950701    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:18:26.950722 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.950721    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:18:26.950833 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.950732    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc8bw\" (UniqueName: \"kubernetes.io/projected/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-kube-api-access-vc8bw\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:18:26.950833 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:26.950741    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:18:27.055315 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:27.055287    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" path="/var/lib/kubelet/pods/e90fc98c-c2a8-4d8d-8c32-6e9a17f08370/volumes"
Apr 24 23:18:35.907035 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.907000    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"]
Apr 24 23:18:35.907567 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.907326    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy" containerID="cri-o://50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5" gracePeriod=30
Apr 24 23:18:35.907567 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.907308    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kserve-container" containerID="cri-o://96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098" gracePeriod=30
Apr 24 23:18:35.937781 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.937757    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"]
Apr 24 23:18:35.938089 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.938072    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="storage-initializer"
Apr 24 23:18:35.938166 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.938092    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="storage-initializer"
Apr 24 23:18:35.938166 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.938116    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy"
Apr 24 23:18:35.938166 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.938125    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy"
Apr 24 23:18:35.938166 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.938135    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kserve-container"
Apr 24 23:18:35.938166 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.938144    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kserve-container"
Apr 24 23:18:35.938454 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.938234    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kube-rbac-proxy"
Apr 24 23:18:35.938454 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.938249    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="e90fc98c-c2a8-4d8d-8c32-6e9a17f08370" containerName="kserve-container"
Apr 24 23:18:35.942757 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.942740    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:35.945199 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.945180    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Apr 24 23:18:35.945281 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.945180    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Apr 24 23:18:35.951009 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:35.950982    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"]
Apr 24 23:18:36.014957 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.014932    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdbf155-172c-430c-a912-792047b69bfc-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.015092 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.014993    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdbf155-172c-430c-a912-792047b69bfc-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.015092 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.015076    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5ks\" (UniqueName: \"kubernetes.io/projected/1fdbf155-172c-430c-a912-792047b69bfc-kube-api-access-vq5ks\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.015188 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.015120    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdbf155-172c-430c-a912-792047b69bfc-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.116150 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.116127    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdbf155-172c-430c-a912-792047b69bfc-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.116287 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.116160    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5ks\" (UniqueName: \"kubernetes.io/projected/1fdbf155-172c-430c-a912-792047b69bfc-kube-api-access-vq5ks\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.116287 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.116179    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdbf155-172c-430c-a912-792047b69bfc-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.116287 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.116224    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdbf155-172c-430c-a912-792047b69bfc-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.116565 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.116543    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdbf155-172c-430c-a912-792047b69bfc-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.116869 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.116847    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdbf155-172c-430c-a912-792047b69bfc-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.118605 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.118587    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdbf155-172c-430c-a912-792047b69bfc-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.124882 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.124860    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5ks\" (UniqueName: \"kubernetes.io/projected/1fdbf155-172c-430c-a912-792047b69bfc-kube-api-access-vq5ks\") pod \"isvc-triton-predictor-84bb65d94b-w8r97\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.252684 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.252657    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:18:36.372396 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.372254    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"]
Apr 24 23:18:36.374426 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:18:36.374398    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdbf155_172c_430c_a912_792047b69bfc.slice/crio-f714a5bc5fc4e576a58ec9952fd9ac1755d9bf56f93b55aa73756bd2762fe365 WatchSource:0}: Error finding container f714a5bc5fc4e576a58ec9952fd9ac1755d9bf56f93b55aa73756bd2762fe365: Status 404 returned error can't find the container with id f714a5bc5fc4e576a58ec9952fd9ac1755d9bf56f93b55aa73756bd2762fe365
Apr 24 23:18:36.936873 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.936842    2543 generic.go:358] "Generic (PLEG): container finished" podID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerID="50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5" exitCode=2
Apr 24 23:18:36.937240 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.936909    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" event={"ID":"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb","Type":"ContainerDied","Data":"50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5"}
Apr 24 23:18:36.938132 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.938107    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" event={"ID":"1fdbf155-172c-430c-a912-792047b69bfc","Type":"ContainerStarted","Data":"a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702"}
Apr 24 23:18:36.938132 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:36.938137    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" event={"ID":"1fdbf155-172c-430c-a912-792047b69bfc","Type":"ContainerStarted","Data":"f714a5bc5fc4e576a58ec9952fd9ac1755d9bf56f93b55aa73756bd2762fe365"}
Apr 24 23:18:39.845380 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:39.845324    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Apr 24 23:18:40.950334 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:40.950300    2543 generic.go:358] "Generic (PLEG): container finished" podID="1fdbf155-172c-430c-a912-792047b69bfc" containerID="a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702" exitCode=0
Apr 24 23:18:40.950698 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:40.950382    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" event={"ID":"1fdbf155-172c-430c-a912-792047b69bfc","Type":"ContainerDied","Data":"a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702"}
Apr 24 23:18:44.846353 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:44.845926    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Apr 24 23:18:49.845510 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:49.845458    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Apr 24 23:18:49.846079 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:49.845633    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:18:54.846104 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:54.846057    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Apr 24 23:18:59.846253 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:18:59.846150    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Apr 24 23:19:04.845669 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:04.845622    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Apr 24 23:19:06.632971 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.632944    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:19:06.790855 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.790743    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbzdh\" (UniqueName: \"kubernetes.io/projected/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kube-api-access-gbzdh\") pod \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") "
Apr 24 23:19:06.790855 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.790821    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-proxy-tls\") pod \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") "
Apr 24 23:19:06.790855 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.790852    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") "
Apr 24 23:19:06.791141 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.790907    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kserve-provision-location\") pod \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\" (UID: \"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb\") "
Apr 24 23:19:06.791550 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.791516    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config") pod "a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" (UID: "a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb"). InnerVolumeSpecName "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:19:06.793583 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.793542    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" (UID: "a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:19:06.793893 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.793864    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kube-api-access-gbzdh" (OuterVolumeSpecName: "kube-api-access-gbzdh") pod "a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" (UID: "a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb"). InnerVolumeSpecName "kube-api-access-gbzdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:19:06.795833 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.795810    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" (UID: "a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:19:06.892476 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.892444    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzdh\" (UniqueName: \"kubernetes.io/projected/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kube-api-access-gbzdh\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:19:06.892476 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.892479    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:19:06.892678 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.892495    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:19:06.892678 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:06.892512    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:19:07.065137 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.065050    2543 generic.go:358] "Generic (PLEG): container finished" podID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerID="96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098" exitCode=137
Apr 24 23:19:07.065297 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.065145    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" event={"ID":"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb","Type":"ContainerDied","Data":"96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098"}
Apr 24 23:19:07.065297 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.065180    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w" event={"ID":"a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb","Type":"ContainerDied","Data":"b882bba716cb1be13be9f3bd2ecfd13b6d9b2f3b0c0316d3073c7afc90b089d3"}
Apr 24 23:19:07.065297 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.065202    2543 scope.go:117] "RemoveContainer" containerID="50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5"
Apr 24 23:19:07.065297 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.065155    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"
Apr 24 23:19:07.077556 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.077524    2543 scope.go:117] "RemoveContainer" containerID="96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098"
Apr 24 23:19:07.087140 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.087122    2543 scope.go:117] "RemoveContainer" containerID="d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd"
Apr 24 23:19:07.090440 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.090416    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"]
Apr 24 23:19:07.092427 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.092406    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-vn98w"]
Apr 24 23:19:07.097281 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.097262    2543 scope.go:117] "RemoveContainer" containerID="50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5"
Apr 24 23:19:07.097646 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:19:07.097616    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5\": container with ID starting with 50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5 not found: ID does not exist" containerID="50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5"
Apr 24 23:19:07.097764 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.097651    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5"} err="failed to get container status \"50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5\": rpc error: code = NotFound desc = could not find container \"50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5\": container with ID starting with 50382a8cdd361ec24ffccfed114f22429fd79c207d7b7b72618d373f077da1b5 not found: ID does not exist"
Apr 24 23:19:07.097764 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.097677    2543 scope.go:117] "RemoveContainer" containerID="96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098"
Apr 24 23:19:07.097968 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:19:07.097942    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098\": container with ID starting with 96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098 not found: ID does not exist" containerID="96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098"
Apr 24 23:19:07.098067 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.097978    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098"} err="failed to get container status \"96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098\": rpc error: code = NotFound desc = could not find container \"96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098\": container with ID starting with 96d0c5f992992df704791f88c58eb32a5c3ea6da8b39835ba5e3a7c4e1630098 not found: ID does not exist"
Apr 24 23:19:07.098067 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.098002    2543 scope.go:117] "RemoveContainer" containerID="d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd"
Apr 24 23:19:07.098485 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:19:07.098461    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd\": container with ID starting with d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd not found: ID does not exist" containerID="d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd"
Apr 24 23:19:07.098557 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:07.098495    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd"} err="failed to get container status \"d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd\": rpc error: code = NotFound desc = could not find container \"d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd\": container with ID starting with d50a06f6abaccd182bcbe0e82d7ffc3772a315806c0bb52de5c6815eb58456dd not found: ID does not exist"
Apr 24 23:19:09.056609 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:19:09.056566    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" path="/var/lib/kubelet/pods/a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb/volumes"
Apr 24 23:20:34.910093 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:34.910056    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:20:34.910619 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:34.910197    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:20:36.337626 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:36.337591    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" event={"ID":"1fdbf155-172c-430c-a912-792047b69bfc","Type":"ContainerStarted","Data":"a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c"}
Apr 24 23:20:36.337626 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:36.337625    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" event={"ID":"1fdbf155-172c-430c-a912-792047b69bfc","Type":"ContainerStarted","Data":"370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef"}
Apr 24 23:20:36.338093 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:36.337793    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:20:36.338093 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:36.337834    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:20:36.338927 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:36.338905    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Apr 24 23:20:36.357451 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:36.357400    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" podStartSLOduration=6.560187908 podStartE2EDuration="2m1.357385588s" podCreationTimestamp="2026-04-24 23:18:35 +0000 UTC" firstStartedPulling="2026-04-24 23:18:40.951320549 +0000 UTC m=+2924.377718056" lastFinishedPulling="2026-04-24 23:20:35.748518223 +0000 UTC m=+3039.174915736" observedRunningTime="2026-04-24 23:20:36.35498679 +0000 UTC m=+3039.781384320" watchObservedRunningTime="2026-04-24 23:20:36.357385588 +0000 UTC m=+3039.783783120"
Apr 24 23:20:37.340886 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:37.340833    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Apr 24 23:20:42.345484 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:42.345457    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:20:42.345985 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:42.345896    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Apr 24 23:20:52.346114 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:52.346086    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:20:58.272997 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.272960    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"]
Apr 24 23:20:58.273404 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.273360    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kserve-container" containerID="cri-o://370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef" gracePeriod=30
Apr 24 23:20:58.273480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.273415    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kube-rbac-proxy" containerID="cri-o://a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c" gracePeriod=30
Apr 24 23:20:58.367883 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.367853    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"]
Apr 24 23:20:58.368178 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.368165    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kserve-container"
Apr 24 23:20:58.368227 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.368180    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kserve-container"
Apr 24 23:20:58.368227 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.368198    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy"
Apr 24 23:20:58.368227 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.368204    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy"
Apr 24 23:20:58.368227 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.368213    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="storage-initializer"
Apr 24 23:20:58.368227 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.368220    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="storage-initializer"
Apr 24 23:20:58.368431 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.368263    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kserve-container"
Apr 24 23:20:58.368431 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.368274    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="a3aa846e-1fdf-4cae-8ecc-8916ef89c8bb" containerName="kube-rbac-proxy"
Apr 24 23:20:58.371279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.371264    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.373535 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.373514    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Apr 24 23:20:58.373665 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.373582    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Apr 24 23:20:58.378234 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.378192    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"]
Apr 24 23:20:58.399767 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.399744    2543 generic.go:358] "Generic (PLEG): container finished" podID="1fdbf155-172c-430c-a912-792047b69bfc" containerID="a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c" exitCode=2
Apr 24 23:20:58.399864 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.399812    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" event={"ID":"1fdbf155-172c-430c-a912-792047b69bfc","Type":"ContainerDied","Data":"a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c"}
Apr 24 23:20:58.437140 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.437115    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3052f-8849-4a8a-b614-0eb3c7785b70-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.437227 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.437152    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsjp\" (UniqueName: \"kubernetes.io/projected/21c3052f-8849-4a8a-b614-0eb3c7785b70-kube-api-access-wwsjp\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.437227 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.437169    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3052f-8849-4a8a-b614-0eb3c7785b70-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.437299 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.437247    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3052f-8849-4a8a-b614-0eb3c7785b70-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.538310 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.538253    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3052f-8849-4a8a-b614-0eb3c7785b70-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.538310 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.538302    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3052f-8849-4a8a-b614-0eb3c7785b70-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.538483 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.538330    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsjp\" (UniqueName: \"kubernetes.io/projected/21c3052f-8849-4a8a-b614-0eb3c7785b70-kube-api-access-wwsjp\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.538483 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.538363    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3052f-8849-4a8a-b614-0eb3c7785b70-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.538588 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:20:58.538486    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Apr 24 23:20:58.538588 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:20:58.538546    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21c3052f-8849-4a8a-b614-0eb3c7785b70-proxy-tls podName:21c3052f-8849-4a8a-b614-0eb3c7785b70 nodeName:}" failed. No retries permitted until 2026-04-24 23:20:59.038526919 +0000 UTC m=+3062.464924427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21c3052f-8849-4a8a-b614-0eb3c7785b70-proxy-tls") pod "isvc-xgboost-predictor-8689c4cfcc-n7sd5" (UID: "21c3052f-8849-4a8a-b614-0eb3c7785b70") : secret "isvc-xgboost-predictor-serving-cert" not found
Apr 24 23:20:58.538682 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.538634    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3052f-8849-4a8a-b614-0eb3c7785b70-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.538966 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.538948    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3052f-8849-4a8a-b614-0eb3c7785b70-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:58.548708 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:58.548686    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsjp\" (UniqueName: \"kubernetes.io/projected/21c3052f-8849-4a8a-b614-0eb3c7785b70-kube-api-access-wwsjp\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:59.043331 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:59.043295    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3052f-8849-4a8a-b614-0eb3c7785b70-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:59.045835 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:59.045817    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3052f-8849-4a8a-b614-0eb3c7785b70-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-n7sd5\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:59.281807 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:59.281780    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:20:59.583735 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:59.583712    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"]
Apr 24 23:20:59.593748 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:20:59.593727    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 23:21:00.406506 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.406472    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" event={"ID":"21c3052f-8849-4a8a-b614-0eb3c7785b70","Type":"ContainerStarted","Data":"85f47b10eb072013e7933e3f20c28da3aa0d3a75d5655fe141b8ae8b111f3a57"}
Apr 24 23:21:00.406506 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.406510    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" event={"ID":"21c3052f-8849-4a8a-b614-0eb3c7785b70","Type":"ContainerStarted","Data":"ea4d6c1140a2d3908ee4a7682e1e9543851bab9f018930ecb731ad1bbc54447c"}
Apr 24 23:21:00.709577 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.709555    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:21:00.859276 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.859251    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdbf155-172c-430c-a912-792047b69bfc-isvc-triton-kube-rbac-proxy-sar-config\") pod \"1fdbf155-172c-430c-a912-792047b69bfc\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") "
Apr 24 23:21:00.859449 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.859309    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdbf155-172c-430c-a912-792047b69bfc-proxy-tls\") pod \"1fdbf155-172c-430c-a912-792047b69bfc\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") "
Apr 24 23:21:00.859449 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.859361    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdbf155-172c-430c-a912-792047b69bfc-kserve-provision-location\") pod \"1fdbf155-172c-430c-a912-792047b69bfc\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") "
Apr 24 23:21:00.859449 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.859381    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq5ks\" (UniqueName: \"kubernetes.io/projected/1fdbf155-172c-430c-a912-792047b69bfc-kube-api-access-vq5ks\") pod \"1fdbf155-172c-430c-a912-792047b69bfc\" (UID: \"1fdbf155-172c-430c-a912-792047b69bfc\") "
Apr 24 23:21:00.859719 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.859693    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fdbf155-172c-430c-a912-792047b69bfc-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "1fdbf155-172c-430c-a912-792047b69bfc" (UID: "1fdbf155-172c-430c-a912-792047b69bfc"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:21:00.859759 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.859740    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdbf155-172c-430c-a912-792047b69bfc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1fdbf155-172c-430c-a912-792047b69bfc" (UID: "1fdbf155-172c-430c-a912-792047b69bfc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:21:00.861564 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.861547    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdbf155-172c-430c-a912-792047b69bfc-kube-api-access-vq5ks" (OuterVolumeSpecName: "kube-api-access-vq5ks") pod "1fdbf155-172c-430c-a912-792047b69bfc" (UID: "1fdbf155-172c-430c-a912-792047b69bfc"). InnerVolumeSpecName "kube-api-access-vq5ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:21:00.861643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.861627    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fdbf155-172c-430c-a912-792047b69bfc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1fdbf155-172c-430c-a912-792047b69bfc" (UID: "1fdbf155-172c-430c-a912-792047b69bfc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:21:00.960459 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.960432    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fdbf155-172c-430c-a912-792047b69bfc-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:21:00.960459 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.960459    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdbf155-172c-430c-a912-792047b69bfc-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:21:00.960629 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.960469    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq5ks\" (UniqueName: \"kubernetes.io/projected/1fdbf155-172c-430c-a912-792047b69bfc-kube-api-access-vq5ks\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:21:00.960629 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:00.960481    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fdbf155-172c-430c-a912-792047b69bfc-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:21:01.410874 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.410791    2543 generic.go:358] "Generic (PLEG): container finished" podID="1fdbf155-172c-430c-a912-792047b69bfc" containerID="370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef" exitCode=0
Apr 24 23:21:01.410874 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.410869    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"
Apr 24 23:21:01.411375 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.410870    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" event={"ID":"1fdbf155-172c-430c-a912-792047b69bfc","Type":"ContainerDied","Data":"370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef"}
Apr 24 23:21:01.411375 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.410912    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97" event={"ID":"1fdbf155-172c-430c-a912-792047b69bfc","Type":"ContainerDied","Data":"f714a5bc5fc4e576a58ec9952fd9ac1755d9bf56f93b55aa73756bd2762fe365"}
Apr 24 23:21:01.411375 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.410932    2543 scope.go:117] "RemoveContainer" containerID="a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c"
Apr 24 23:21:01.419216 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.419178    2543 scope.go:117] "RemoveContainer" containerID="370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef"
Apr 24 23:21:01.426355 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.426314    2543 scope.go:117] "RemoveContainer" containerID="a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702"
Apr 24 23:21:01.427141 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.427118    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"]
Apr 24 23:21:01.430764 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.430738    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-w8r97"]
Apr 24 23:21:01.434248 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.434230    2543 scope.go:117] "RemoveContainer" containerID="a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c"
Apr 24 23:21:01.434511 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:21:01.434493    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c\": container with ID starting with a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c not found: ID does not exist" containerID="a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c"
Apr 24 23:21:01.434566 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.434522    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c"} err="failed to get container status \"a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c\": rpc error: code = NotFound desc = could not find container \"a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c\": container with ID starting with a6b6ab09a071d753e2eeec98ddacb708e777751abea0f0da012d2a09deaf147c not found: ID does not exist"
Apr 24 23:21:01.434566 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.434543    2543 scope.go:117] "RemoveContainer" containerID="370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef"
Apr 24 23:21:01.434761 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:21:01.434744    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef\": container with ID starting with 370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef not found: ID does not exist" containerID="370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef"
Apr 24 23:21:01.434811 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.434766    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef"} err="failed to get container status \"370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef\": rpc error: code = NotFound desc = could not find container \"370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef\": container with ID starting with 370febd93a6b6b0dbb41b731dcfa6f14c2f8f6e66abede6e79658327a17209ef not found: ID does not exist"
Apr 24 23:21:01.434811 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.434779    2543 scope.go:117] "RemoveContainer" containerID="a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702"
Apr 24 23:21:01.434998 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:21:01.434984    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702\": container with ID starting with a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702 not found: ID does not exist" containerID="a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702"
Apr 24 23:21:01.435043 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:01.435002    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702"} err="failed to get container status \"a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702\": rpc error: code = NotFound desc = could not find container \"a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702\": container with ID starting with a686c14627d2b3b0bb4ec4fcc4d08a33825d9f8c23545dc4ce6cdab3de038702 not found: ID does not exist"
Apr 24 23:21:03.055644 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:03.055612    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fdbf155-172c-430c-a912-792047b69bfc" path="/var/lib/kubelet/pods/1fdbf155-172c-430c-a912-792047b69bfc/volumes"
Apr 24 23:21:03.418653 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:03.418583    2543 generic.go:358] "Generic (PLEG): container finished" podID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerID="85f47b10eb072013e7933e3f20c28da3aa0d3a75d5655fe141b8ae8b111f3a57" exitCode=0
Apr 24 23:21:03.418904 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:03.418653    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" event={"ID":"21c3052f-8849-4a8a-b614-0eb3c7785b70","Type":"ContainerDied","Data":"85f47b10eb072013e7933e3f20c28da3aa0d3a75d5655fe141b8ae8b111f3a57"}
Apr 24 23:21:22.479716 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:22.479680    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" event={"ID":"21c3052f-8849-4a8a-b614-0eb3c7785b70","Type":"ContainerStarted","Data":"f715a2e18912e925f56450526083b482a091b282739d4a7b68b9d47e232f700c"}
Apr 24 23:21:22.480094 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:22.479723    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" event={"ID":"21c3052f-8849-4a8a-b614-0eb3c7785b70","Type":"ContainerStarted","Data":"c30dad317936cd70e6a68114c46fe7e7b9f0e38dc5e23f1b036f124d66b32984"}
Apr 24 23:21:22.480094 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:22.479923    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:21:22.497434 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:22.497391    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podStartSLOduration=5.898828988 podStartE2EDuration="24.497377096s" podCreationTimestamp="2026-04-24 23:20:58 +0000 UTC" firstStartedPulling="2026-04-24 23:21:03.419922669 +0000 UTC m=+3066.846320177" lastFinishedPulling="2026-04-24 23:21:22.018470777 +0000 UTC m=+3085.444868285" observedRunningTime="2026-04-24 23:21:22.496470518 +0000 UTC m=+3085.922868047" watchObservedRunningTime="2026-04-24 23:21:22.497377096 +0000 UTC m=+3085.923774699"
Apr 24 23:21:23.483144 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:23.483106    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:21:23.484353 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:23.484312    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Apr 24 23:21:24.485816 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:24.485777    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Apr 24 23:21:29.489885 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:29.489857    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:21:29.490478 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:29.490450    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Apr 24 23:21:39.491425 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:39.491389    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Apr 24 23:21:49.490461 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:49.490419    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Apr 24 23:21:59.490499 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:21:59.490456    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Apr 24 23:22:09.491302 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:09.491218    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Apr 24 23:22:19.490481 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:19.490453    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:22:28.466236 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.466205    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"]
Apr 24 23:22:28.466781 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.466594    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container" containerID="cri-o://c30dad317936cd70e6a68114c46fe7e7b9f0e38dc5e23f1b036f124d66b32984" gracePeriod=30
Apr 24 23:22:28.466781 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.466599    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kube-rbac-proxy" containerID="cri-o://f715a2e18912e925f56450526083b482a091b282739d4a7b68b9d47e232f700c" gracePeriod=30
Apr 24 23:22:28.586924 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.586899    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"]
Apr 24 23:22:28.587191 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.587178    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kube-rbac-proxy"
Apr 24 23:22:28.587191 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.587192    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kube-rbac-proxy"
Apr 24 23:22:28.587279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.587200    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="storage-initializer"
Apr 24 23:22:28.587279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.587206    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="storage-initializer"
Apr 24 23:22:28.587279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.587217    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kserve-container"
Apr 24 23:22:28.587279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.587222    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kserve-container"
Apr 24 23:22:28.587279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.587276    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kserve-container"
Apr 24 23:22:28.587520 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.587286    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="1fdbf155-172c-430c-a912-792047b69bfc" containerName="kube-rbac-proxy"
Apr 24 23:22:28.590117 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.590101    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.592494 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.592474    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Apr 24 23:22:28.592590 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.592565    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Apr 24 23:22:28.598231 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.598209    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"]
Apr 24 23:22:28.668668 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.668635    2543 generic.go:358] "Generic (PLEG): container finished" podID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerID="f715a2e18912e925f56450526083b482a091b282739d4a7b68b9d47e232f700c" exitCode=2
Apr 24 23:22:28.668773 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.668711    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" event={"ID":"21c3052f-8849-4a8a-b614-0eb3c7785b70","Type":"ContainerDied","Data":"f715a2e18912e925f56450526083b482a091b282739d4a7b68b9d47e232f700c"}
Apr 24 23:22:28.716099 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.716077    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80c224f-5309-40da-87b9-4016c7bb5675-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.716229 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.716117    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80c224f-5309-40da-87b9-4016c7bb5675-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.716229 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.716138    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80c224f-5309-40da-87b9-4016c7bb5675-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.716229 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.716160    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5kcd\" (UniqueName: \"kubernetes.io/projected/a80c224f-5309-40da-87b9-4016c7bb5675-kube-api-access-z5kcd\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.817549 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.817463    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80c224f-5309-40da-87b9-4016c7bb5675-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.817549 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.817511    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80c224f-5309-40da-87b9-4016c7bb5675-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.817733 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.817548    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z5kcd\" (UniqueName: \"kubernetes.io/projected/a80c224f-5309-40da-87b9-4016c7bb5675-kube-api-access-z5kcd\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.817733 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.817653    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80c224f-5309-40da-87b9-4016c7bb5675-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.818067 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.818031    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80c224f-5309-40da-87b9-4016c7bb5675-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.818229 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.818208    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80c224f-5309-40da-87b9-4016c7bb5675-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.820120 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.820102    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80c224f-5309-40da-87b9-4016c7bb5675-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.825765 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.825746    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5kcd\" (UniqueName: \"kubernetes.io/projected/a80c224f-5309-40da-87b9-4016c7bb5675-kube-api-access-z5kcd\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:28.900116 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:28.900070    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:29.024910 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:29.024883    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"]
Apr 24 23:22:29.027230 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:22:29.027206    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80c224f_5309_40da_87b9_4016c7bb5675.slice/crio-8f6c35055619e4b47e4a95798d07b724c025c021e7b007524bd8d8ec017f2501 WatchSource:0}: Error finding container 8f6c35055619e4b47e4a95798d07b724c025c021e7b007524bd8d8ec017f2501: Status 404 returned error can't find the container with id 8f6c35055619e4b47e4a95798d07b724c025c021e7b007524bd8d8ec017f2501
Apr 24 23:22:29.486675 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:29.486630    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Apr 24 23:22:29.491013 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:29.490984    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Apr 24 23:22:29.672482 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:29.672445    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" event={"ID":"a80c224f-5309-40da-87b9-4016c7bb5675","Type":"ContainerStarted","Data":"ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66"}
Apr 24 23:22:29.672482 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:29.672483    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" event={"ID":"a80c224f-5309-40da-87b9-4016c7bb5675","Type":"ContainerStarted","Data":"8f6c35055619e4b47e4a95798d07b724c025c021e7b007524bd8d8ec017f2501"}
Apr 24 23:22:31.680443 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.680411    2543 generic.go:358] "Generic (PLEG): container finished" podID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerID="c30dad317936cd70e6a68114c46fe7e7b9f0e38dc5e23f1b036f124d66b32984" exitCode=0
Apr 24 23:22:31.680768 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.680469    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" event={"ID":"21c3052f-8849-4a8a-b614-0eb3c7785b70","Type":"ContainerDied","Data":"c30dad317936cd70e6a68114c46fe7e7b9f0e38dc5e23f1b036f124d66b32984"}
Apr 24 23:22:31.701974 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.701956    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:22:31.840483 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.840395    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3052f-8849-4a8a-b614-0eb3c7785b70-proxy-tls\") pod \"21c3052f-8849-4a8a-b614-0eb3c7785b70\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") "
Apr 24 23:22:31.840483 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.840449    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3052f-8849-4a8a-b614-0eb3c7785b70-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"21c3052f-8849-4a8a-b614-0eb3c7785b70\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") "
Apr 24 23:22:31.840696 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.840517    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwsjp\" (UniqueName: \"kubernetes.io/projected/21c3052f-8849-4a8a-b614-0eb3c7785b70-kube-api-access-wwsjp\") pod \"21c3052f-8849-4a8a-b614-0eb3c7785b70\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") "
Apr 24 23:22:31.840696 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.840554    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3052f-8849-4a8a-b614-0eb3c7785b70-kserve-provision-location\") pod \"21c3052f-8849-4a8a-b614-0eb3c7785b70\" (UID: \"21c3052f-8849-4a8a-b614-0eb3c7785b70\") "
Apr 24 23:22:31.840786 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.840749    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c3052f-8849-4a8a-b614-0eb3c7785b70-isvc-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-kube-rbac-proxy-sar-config") pod "21c3052f-8849-4a8a-b614-0eb3c7785b70" (UID: "21c3052f-8849-4a8a-b614-0eb3c7785b70"). InnerVolumeSpecName "isvc-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:22:31.840927 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.840898    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c3052f-8849-4a8a-b614-0eb3c7785b70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21c3052f-8849-4a8a-b614-0eb3c7785b70" (UID: "21c3052f-8849-4a8a-b614-0eb3c7785b70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:22:31.842643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.842615    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c3052f-8849-4a8a-b614-0eb3c7785b70-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21c3052f-8849-4a8a-b614-0eb3c7785b70" (UID: "21c3052f-8849-4a8a-b614-0eb3c7785b70"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:22:31.842643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.842623    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c3052f-8849-4a8a-b614-0eb3c7785b70-kube-api-access-wwsjp" (OuterVolumeSpecName: "kube-api-access-wwsjp") pod "21c3052f-8849-4a8a-b614-0eb3c7785b70" (UID: "21c3052f-8849-4a8a-b614-0eb3c7785b70"). InnerVolumeSpecName "kube-api-access-wwsjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:22:31.941143 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.941121    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21c3052f-8849-4a8a-b614-0eb3c7785b70-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:22:31.941143 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.941142    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21c3052f-8849-4a8a-b614-0eb3c7785b70-isvc-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:22:31.941283 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.941152    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwsjp\" (UniqueName: \"kubernetes.io/projected/21c3052f-8849-4a8a-b614-0eb3c7785b70-kube-api-access-wwsjp\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:22:31.941283 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:31.941161    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c3052f-8849-4a8a-b614-0eb3c7785b70-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:22:32.685062 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:32.685029    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5" event={"ID":"21c3052f-8849-4a8a-b614-0eb3c7785b70","Type":"ContainerDied","Data":"ea4d6c1140a2d3908ee4a7682e1e9543851bab9f018930ecb731ad1bbc54447c"}
Apr 24 23:22:32.685062 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:32.685043    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"
Apr 24 23:22:32.685540 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:32.685071    2543 scope.go:117] "RemoveContainer" containerID="f715a2e18912e925f56450526083b482a091b282739d4a7b68b9d47e232f700c"
Apr 24 23:22:32.693736 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:32.693712    2543 scope.go:117] "RemoveContainer" containerID="c30dad317936cd70e6a68114c46fe7e7b9f0e38dc5e23f1b036f124d66b32984"
Apr 24 23:22:32.701105 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:32.701088    2543 scope.go:117] "RemoveContainer" containerID="85f47b10eb072013e7933e3f20c28da3aa0d3a75d5655fe141b8ae8b111f3a57"
Apr 24 23:22:32.707236 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:32.707207    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"]
Apr 24 23:22:32.712763 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:32.712743    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-n7sd5"]
Apr 24 23:22:33.054670 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:33.054643    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" path="/var/lib/kubelet/pods/21c3052f-8849-4a8a-b614-0eb3c7785b70/volumes"
Apr 24 23:22:33.690525 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:33.690489    2543 generic.go:358] "Generic (PLEG): container finished" podID="a80c224f-5309-40da-87b9-4016c7bb5675" containerID="ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66" exitCode=0
Apr 24 23:22:33.690879 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:33.690566    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" event={"ID":"a80c224f-5309-40da-87b9-4016c7bb5675","Type":"ContainerDied","Data":"ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66"}
Apr 24 23:22:34.695502 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:34.695469    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" event={"ID":"a80c224f-5309-40da-87b9-4016c7bb5675","Type":"ContainerStarted","Data":"ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7"}
Apr 24 23:22:34.695874 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:34.695513    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" event={"ID":"a80c224f-5309-40da-87b9-4016c7bb5675","Type":"ContainerStarted","Data":"0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f"}
Apr 24 23:22:34.695874 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:34.695738    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:34.715209 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:34.715167    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" podStartSLOduration=6.715155688 podStartE2EDuration="6.715155688s" podCreationTimestamp="2026-04-24 23:22:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:22:34.713489949 +0000 UTC m=+3158.139887489" watchObservedRunningTime="2026-04-24 23:22:34.715155688 +0000 UTC m=+3158.141553218"
Apr 24 23:22:35.698267 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:35.698241    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:22:41.707178 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:22:41.707148    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:23:11.711435 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:11.711406    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:23:18.634717 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.634686    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"]
Apr 24 23:23:18.635180 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.635089    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kserve-container" containerID="cri-o://0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f" gracePeriod=30
Apr 24 23:23:18.635249 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.635172    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kube-rbac-proxy" containerID="cri-o://ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7" gracePeriod=30
Apr 24 23:23:18.728589 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.728558    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"]
Apr 24 23:23:18.728876 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.728863    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kube-rbac-proxy"
Apr 24 23:23:18.728928 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.728877    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kube-rbac-proxy"
Apr 24 23:23:18.728928 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.728890    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="storage-initializer"
Apr 24 23:23:18.728928 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.728896    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="storage-initializer"
Apr 24 23:23:18.728928 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.728902    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container"
Apr 24 23:23:18.728928 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.728907    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container"
Apr 24 23:23:18.729106 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.728957    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kube-rbac-proxy"
Apr 24 23:23:18.729106 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.728966    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="21c3052f-8849-4a8a-b614-0eb3c7785b70" containerName="kserve-container"
Apr 24 23:23:18.733261 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.733244    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.735568 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.735546    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Apr 24 23:23:18.735742 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.735583    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Apr 24 23:23:18.740439 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.740411    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"]
Apr 24 23:23:18.829984 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.829958    2543 generic.go:358] "Generic (PLEG): container finished" podID="a80c224f-5309-40da-87b9-4016c7bb5675" containerID="ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7" exitCode=2
Apr 24 23:23:18.830120 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.830020    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" event={"ID":"a80c224f-5309-40da-87b9-4016c7bb5675","Type":"ContainerDied","Data":"ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7"}
Apr 24 23:23:18.875265 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.875239    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.875427 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.875279    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpch\" (UniqueName: \"kubernetes.io/projected/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kube-api-access-fmpch\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.875427 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.875404    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.875547 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.875446    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.976252 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.976217    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.976406 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.976269    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpch\" (UniqueName: \"kubernetes.io/projected/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kube-api-access-fmpch\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.976406 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.976371    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.976406 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.976400    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.976666 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.976642    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.977013 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.976992    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.978859 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.978839    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:18.984488 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:18.984465    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpch\" (UniqueName: \"kubernetes.io/projected/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kube-api-access-fmpch\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-6grg2\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:19.044315 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:19.044291    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:19.163779 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:19.163744    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"]
Apr 24 23:23:19.167432 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:23:19.167397    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d2103a_267a_4f0b_b961_bcf99ae2a1bd.slice/crio-f3510646aa199c8d1495cee51ea2fb20ecf035068c86f3e1f17c7201db9e52b4 WatchSource:0}: Error finding container f3510646aa199c8d1495cee51ea2fb20ecf035068c86f3e1f17c7201db9e52b4: Status 404 returned error can't find the container with id f3510646aa199c8d1495cee51ea2fb20ecf035068c86f3e1f17c7201db9e52b4
Apr 24 23:23:19.835140 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:19.835100    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" event={"ID":"67d2103a-267a-4f0b-b961-bcf99ae2a1bd","Type":"ContainerStarted","Data":"3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8"}
Apr 24 23:23:19.835140 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:19.835138    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" event={"ID":"67d2103a-267a-4f0b-b961-bcf99ae2a1bd","Type":"ContainerStarted","Data":"f3510646aa199c8d1495cee51ea2fb20ecf035068c86f3e1f17c7201db9e52b4"}
Apr 24 23:23:21.702902 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:21.702863    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Apr 24 23:23:21.707770 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:21.707746    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.56:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.56:8080: connect: connection refused"
Apr 24 23:23:22.846008 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:22.845981    2543 generic.go:358] "Generic (PLEG): container finished" podID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerID="3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8" exitCode=0
Apr 24 23:23:22.846288 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:22.846049    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" event={"ID":"67d2103a-267a-4f0b-b961-bcf99ae2a1bd","Type":"ContainerDied","Data":"3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8"}
Apr 24 23:23:23.850759 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:23.850727    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" event={"ID":"67d2103a-267a-4f0b-b961-bcf99ae2a1bd","Type":"ContainerStarted","Data":"035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1"}
Apr 24 23:23:23.850759 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:23.850763    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" event={"ID":"67d2103a-267a-4f0b-b961-bcf99ae2a1bd","Type":"ContainerStarted","Data":"92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0"}
Apr 24 23:23:23.851175 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:23.851073    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:23.851175 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:23.851102    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:23.872600 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:23.872559    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" podStartSLOduration=5.872546467 podStartE2EDuration="5.872546467s" podCreationTimestamp="2026-04-24 23:23:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:23:23.870333805 +0000 UTC m=+3207.296731334" watchObservedRunningTime="2026-04-24 23:23:23.872546467 +0000 UTC m=+3207.298943997"
Apr 24 23:23:24.772008 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.771984    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:23:24.854754 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.854717    2543 generic.go:358] "Generic (PLEG): container finished" podID="a80c224f-5309-40da-87b9-4016c7bb5675" containerID="0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f" exitCode=0
Apr 24 23:23:24.855090 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.854756    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" event={"ID":"a80c224f-5309-40da-87b9-4016c7bb5675","Type":"ContainerDied","Data":"0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f"}
Apr 24 23:23:24.855090 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.854794    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n" event={"ID":"a80c224f-5309-40da-87b9-4016c7bb5675","Type":"ContainerDied","Data":"8f6c35055619e4b47e4a95798d07b724c025c021e7b007524bd8d8ec017f2501"}
Apr 24 23:23:24.855090 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.854811    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"
Apr 24 23:23:24.855090 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.854815    2543 scope.go:117] "RemoveContainer" containerID="ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7"
Apr 24 23:23:24.862672 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.862656    2543 scope.go:117] "RemoveContainer" containerID="0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f"
Apr 24 23:23:24.869560 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.869544    2543 scope.go:117] "RemoveContainer" containerID="ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66"
Apr 24 23:23:24.876033 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.876017    2543 scope.go:117] "RemoveContainer" containerID="ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7"
Apr 24 23:23:24.876240 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:23:24.876223    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7\": container with ID starting with ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7 not found: ID does not exist" containerID="ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7"
Apr 24 23:23:24.876289 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.876246    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7"} err="failed to get container status \"ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7\": rpc error: code = NotFound desc = could not find container \"ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7\": container with ID starting with ffb869956f302b3a2c69fc7912a626ee98562f2e7d1c6ce74e5521ae023beac7 not found: ID does not exist"
Apr 24 23:23:24.876289 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.876264    2543 scope.go:117] "RemoveContainer" containerID="0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f"
Apr 24 23:23:24.876524 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:23:24.876506    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f\": container with ID starting with 0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f not found: ID does not exist" containerID="0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f"
Apr 24 23:23:24.876568 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.876531    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f"} err="failed to get container status \"0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f\": rpc error: code = NotFound desc = could not find container \"0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f\": container with ID starting with 0281a64585c8a7f38065df93f162945a88cc90f5a0e3087fd0bc449b4ebbfe7f not found: ID does not exist"
Apr 24 23:23:24.876568 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.876547    2543 scope.go:117] "RemoveContainer" containerID="ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66"
Apr 24 23:23:24.876784 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:23:24.876766    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66\": container with ID starting with ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66 not found: ID does not exist" containerID="ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66"
Apr 24 23:23:24.876825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.876790    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66"} err="failed to get container status \"ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66\": rpc error: code = NotFound desc = could not find container \"ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66\": container with ID starting with ca95ca60a6b66f7053a98eab6fb784e3e88396992c82f39e1246661f2b6b9d66 not found: ID does not exist"
Apr 24 23:23:24.925295 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.925275    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5kcd\" (UniqueName: \"kubernetes.io/projected/a80c224f-5309-40da-87b9-4016c7bb5675-kube-api-access-z5kcd\") pod \"a80c224f-5309-40da-87b9-4016c7bb5675\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") "
Apr 24 23:23:24.925408 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.925320    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80c224f-5309-40da-87b9-4016c7bb5675-kserve-provision-location\") pod \"a80c224f-5309-40da-87b9-4016c7bb5675\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") "
Apr 24 23:23:24.925408 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.925364    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80c224f-5309-40da-87b9-4016c7bb5675-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"a80c224f-5309-40da-87b9-4016c7bb5675\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") "
Apr 24 23:23:24.925520 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.925430    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80c224f-5309-40da-87b9-4016c7bb5675-proxy-tls\") pod \"a80c224f-5309-40da-87b9-4016c7bb5675\" (UID: \"a80c224f-5309-40da-87b9-4016c7bb5675\") "
Apr 24 23:23:24.925660 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.925637    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a80c224f-5309-40da-87b9-4016c7bb5675-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a80c224f-5309-40da-87b9-4016c7bb5675" (UID: "a80c224f-5309-40da-87b9-4016c7bb5675"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:23:24.925740 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.925724    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a80c224f-5309-40da-87b9-4016c7bb5675-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:23:24.925822 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.925740    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c224f-5309-40da-87b9-4016c7bb5675-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "a80c224f-5309-40da-87b9-4016c7bb5675" (UID: "a80c224f-5309-40da-87b9-4016c7bb5675"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:23:24.927461 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.927424    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80c224f-5309-40da-87b9-4016c7bb5675-kube-api-access-z5kcd" (OuterVolumeSpecName: "kube-api-access-z5kcd") pod "a80c224f-5309-40da-87b9-4016c7bb5675" (UID: "a80c224f-5309-40da-87b9-4016c7bb5675"). InnerVolumeSpecName "kube-api-access-z5kcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:23:24.927570 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:24.927551    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80c224f-5309-40da-87b9-4016c7bb5675-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a80c224f-5309-40da-87b9-4016c7bb5675" (UID: "a80c224f-5309-40da-87b9-4016c7bb5675"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:23:25.026546 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:25.026522    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a80c224f-5309-40da-87b9-4016c7bb5675-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:23:25.026546 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:25.026543    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a80c224f-5309-40da-87b9-4016c7bb5675-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:23:25.026695 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:25.026554    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5kcd\" (UniqueName: \"kubernetes.io/projected/a80c224f-5309-40da-87b9-4016c7bb5675-kube-api-access-z5kcd\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:23:25.179567 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:25.179511    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"]
Apr 24 23:23:25.182761 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:25.182738    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-dzn9n"]
Apr 24 23:23:27.054576 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:27.054548    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" path="/var/lib/kubelet/pods/a80c224f-5309-40da-87b9-4016c7bb5675/volumes"
Apr 24 23:23:29.860405 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:29.860315    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:23:59.863798 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:23:59.863722    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:24:08.815587 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.815556    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"]
Apr 24 23:24:08.816063 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.815866    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kserve-container" containerID="cri-o://92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0" gracePeriod=30
Apr 24 23:24:08.816063 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.815914    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kube-rbac-proxy" containerID="cri-o://035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1" gracePeriod=30
Apr 24 23:24:08.917420 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.917382    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"]
Apr 24 23:24:08.917779 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.917762    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kserve-container"
Apr 24 23:24:08.917859 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.917781    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kserve-container"
Apr 24 23:24:08.917859 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.917799    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kube-rbac-proxy"
Apr 24 23:24:08.917859 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.917807    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kube-rbac-proxy"
Apr 24 23:24:08.917859 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.917827    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="storage-initializer"
Apr 24 23:24:08.917859 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.917836    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="storage-initializer"
Apr 24 23:24:08.918177 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.917913    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kserve-container"
Apr 24 23:24:08.918177 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.917928    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="a80c224f-5309-40da-87b9-4016c7bb5675" containerName="kube-rbac-proxy"
Apr 24 23:24:08.922358 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.922325    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:08.924811 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.924789    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Apr 24 23:24:08.924901 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.924823    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Apr 24 23:24:08.930200 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.930168    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"]
Apr 24 23:24:08.970233 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.970200    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:08.970365 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.970256    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlrgl\" (UniqueName: \"kubernetes.io/projected/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kube-api-access-rlrgl\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:08.970365 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.970320    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:08.970453 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.970387    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:08.982594 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.982567    2543 generic.go:358] "Generic (PLEG): container finished" podID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerID="035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1" exitCode=2
Apr 24 23:24:08.982688 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:08.982636    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" event={"ID":"67d2103a-267a-4f0b-b961-bcf99ae2a1bd","Type":"ContainerDied","Data":"035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1"}
Apr 24 23:24:09.070851 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.070781    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.070851 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.070828    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.071020 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.070868    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.071020 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.070901    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rlrgl\" (UniqueName: \"kubernetes.io/projected/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kube-api-access-rlrgl\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.071020 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:24:09.071006    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Apr 24 23:24:09.071164 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:24:09.071109    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-proxy-tls podName:66bbc8ec-2fe9-4d9e-b320-5279848d64ae nodeName:}" failed. No retries permitted until 2026-04-24 23:24:09.571077431 +0000 UTC m=+3252.997474940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-proxy-tls") pod "isvc-xgboost-runtime-predictor-779db84d9-9kzfb" (UID: "66bbc8ec-2fe9-4d9e-b320-5279848d64ae") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Apr 24 23:24:09.071164 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.071141    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.071568 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.071549    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.079308 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.079287    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlrgl\" (UniqueName: \"kubernetes.io/projected/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kube-api-access-rlrgl\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.574855 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.574819    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.577440 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.577418    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-9kzfb\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.833982 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.833879    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:09.856099 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.856061    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Apr 24 23:24:09.953454 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.953431    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"]
Apr 24 23:24:09.955650 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:24:09.955623    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bbc8ec_2fe9_4d9e_b320_5279848d64ae.slice/crio-1b8002724341f9e97d11fd9de4f59b2e5e502d96c6840b093b973b6d35402711 WatchSource:0}: Error finding container 1b8002724341f9e97d11fd9de4f59b2e5e502d96c6840b093b973b6d35402711: Status 404 returned error can't find the container with id 1b8002724341f9e97d11fd9de4f59b2e5e502d96c6840b093b973b6d35402711
Apr 24 23:24:09.985956 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:09.985928    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" event={"ID":"66bbc8ec-2fe9-4d9e-b320-5279848d64ae","Type":"ContainerStarted","Data":"1b8002724341f9e97d11fd9de4f59b2e5e502d96c6840b093b973b6d35402711"}
Apr 24 23:24:10.901517 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:10.901471    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.57:8080/v2/models/xgboost-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Apr 24 23:24:10.990159 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:10.990126    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" event={"ID":"66bbc8ec-2fe9-4d9e-b320-5279848d64ae","Type":"ContainerStarted","Data":"a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1"}
Apr 24 23:24:14.005973 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:14.005942    2543 generic.go:358] "Generic (PLEG): container finished" podID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerID="a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1" exitCode=0
Apr 24 23:24:14.006494 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:14.006021    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" event={"ID":"66bbc8ec-2fe9-4d9e-b320-5279848d64ae","Type":"ContainerDied","Data":"a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1"}
Apr 24 23:24:14.855715 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:14.855677    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Apr 24 23:24:15.010446 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.010411    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" event={"ID":"66bbc8ec-2fe9-4d9e-b320-5279848d64ae","Type":"ContainerStarted","Data":"3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035"}
Apr 24 23:24:15.010839 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.010454    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" event={"ID":"66bbc8ec-2fe9-4d9e-b320-5279848d64ae","Type":"ContainerStarted","Data":"b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa"}
Apr 24 23:24:15.010839 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.010691    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:15.030720 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.030683    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podStartSLOduration=7.030668716 podStartE2EDuration="7.030668716s" podCreationTimestamp="2026-04-24 23:24:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:24:15.028653046 +0000 UTC m=+3258.455050578" watchObservedRunningTime="2026-04-24 23:24:15.030668716 +0000 UTC m=+3258.457066243"
Apr 24 23:24:15.150280 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.150260    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:24:15.222437 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.222411    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kserve-provision-location\") pod \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") "
Apr 24 23:24:15.222557 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.222468    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") "
Apr 24 23:24:15.222557 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.222494    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-proxy-tls\") pod \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") "
Apr 24 23:24:15.222557 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.222540    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmpch\" (UniqueName: \"kubernetes.io/projected/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kube-api-access-fmpch\") pod \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\" (UID: \"67d2103a-267a-4f0b-b961-bcf99ae2a1bd\") "
Apr 24 23:24:15.222774 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.222748    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67d2103a-267a-4f0b-b961-bcf99ae2a1bd" (UID: "67d2103a-267a-4f0b-b961-bcf99ae2a1bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:24:15.222834 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.222818    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "67d2103a-267a-4f0b-b961-bcf99ae2a1bd" (UID: "67d2103a-267a-4f0b-b961-bcf99ae2a1bd"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:24:15.224649 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.224621    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kube-api-access-fmpch" (OuterVolumeSpecName: "kube-api-access-fmpch") pod "67d2103a-267a-4f0b-b961-bcf99ae2a1bd" (UID: "67d2103a-267a-4f0b-b961-bcf99ae2a1bd"). InnerVolumeSpecName "kube-api-access-fmpch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:24:15.224741 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.224680    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67d2103a-267a-4f0b-b961-bcf99ae2a1bd" (UID: "67d2103a-267a-4f0b-b961-bcf99ae2a1bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:24:15.323164 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.323140    2543 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:24:15.323164 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.323161    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:24:15.323282 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.323171    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmpch\" (UniqueName: \"kubernetes.io/projected/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kube-api-access-fmpch\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:24:15.323282 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:15.323180    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67d2103a-267a-4f0b-b961-bcf99ae2a1bd-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:24:16.014896 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.014861    2543 generic.go:358] "Generic (PLEG): container finished" podID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerID="92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0" exitCode=0
Apr 24 23:24:16.015317 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.014973    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"
Apr 24 23:24:16.015317 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.014996    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" event={"ID":"67d2103a-267a-4f0b-b961-bcf99ae2a1bd","Type":"ContainerDied","Data":"92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0"}
Apr 24 23:24:16.015317 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.015037    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2" event={"ID":"67d2103a-267a-4f0b-b961-bcf99ae2a1bd","Type":"ContainerDied","Data":"f3510646aa199c8d1495cee51ea2fb20ecf035068c86f3e1f17c7201db9e52b4"}
Apr 24 23:24:16.015317 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.015054    2543 scope.go:117] "RemoveContainer" containerID="035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1"
Apr 24 23:24:16.015718 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.015698    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:16.016755 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.016728    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 24 23:24:16.023439 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.023421    2543 scope.go:117] "RemoveContainer" containerID="92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0"
Apr 24 23:24:16.030848 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.030833    2543 scope.go:117] "RemoveContainer" containerID="3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8"
Apr 24 23:24:16.036640 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.036621    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"]
Apr 24 23:24:16.037822 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.037625    2543 scope.go:117] "RemoveContainer" containerID="035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1"
Apr 24 23:24:16.037887 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:24:16.037861    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1\": container with ID starting with 035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1 not found: ID does not exist" containerID="035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1"
Apr 24 23:24:16.037948 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.037883    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1"} err="failed to get container status \"035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1\": rpc error: code = NotFound desc = could not find container \"035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1\": container with ID starting with 035361aebfaf0fa8ac0c7f1a6fe3bc50c5894ef863cffa5f10021883ef0b8bd1 not found: ID does not exist"
Apr 24 23:24:16.037948 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.037901    2543 scope.go:117] "RemoveContainer" containerID="92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0"
Apr 24 23:24:16.038125 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:24:16.038105    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0\": container with ID starting with 92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0 not found: ID does not exist" containerID="92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0"
Apr 24 23:24:16.038191 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.038129    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0"} err="failed to get container status \"92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0\": rpc error: code = NotFound desc = could not find container \"92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0\": container with ID starting with 92fd91ea9576469147caae7a81433bb2b960eab3bba55f972cf0ad2d60505ff0 not found: ID does not exist"
Apr 24 23:24:16.038191 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.038144    2543 scope.go:117] "RemoveContainer" containerID="3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8"
Apr 24 23:24:16.038386 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:24:16.038366    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8\": container with ID starting with 3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8 not found: ID does not exist" containerID="3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8"
Apr 24 23:24:16.038498 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.038389    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8"} err="failed to get container status \"3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8\": rpc error: code = NotFound desc = could not find container \"3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8\": container with ID starting with 3d30d9f3aa269c6e7fd8e2fd573f9bb89c3af106627cd848feaf9634741b64f8 not found: ID does not exist"
Apr 24 23:24:16.042182 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:16.042164    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-6grg2"]
Apr 24 23:24:17.018902 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:17.018863    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 24 23:24:17.055701 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:17.055673    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" path="/var/lib/kubelet/pods/67d2103a-267a-4f0b-b961-bcf99ae2a1bd/volumes"
Apr 24 23:24:22.023809 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:22.023782    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:24:22.024382 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:22.024329    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 24 23:24:32.024264 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:32.024220    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 24 23:24:42.024278 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:42.024237    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 24 23:24:52.024403 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:24:52.024364    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 24 23:25:02.024512 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:02.024473    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 24 23:25:12.025383 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:12.025288    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:25:18.998389 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:18.998333    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"]
Apr 24 23:25:18.998907 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:18.998688    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container" containerID="cri-o://b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa" gracePeriod=30
Apr 24 23:25:18.998907 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:18.998782    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kube-rbac-proxy" containerID="cri-o://3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035" gracePeriod=30
Apr 24 23:25:19.123835 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.123809    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"]
Apr 24 23:25:19.124108 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.124097    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="storage-initializer"
Apr 24 23:25:19.124152 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.124110    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="storage-initializer"
Apr 24 23:25:19.124152 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.124123    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kube-rbac-proxy"
Apr 24 23:25:19.124152 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.124130    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kube-rbac-proxy"
Apr 24 23:25:19.124152 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.124139    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kserve-container"
Apr 24 23:25:19.124152 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.124144    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kserve-container"
Apr 24 23:25:19.124317 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.124200    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kserve-container"
Apr 24 23:25:19.124317 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.124207    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="67d2103a-267a-4f0b-b961-bcf99ae2a1bd" containerName="kube-rbac-proxy"
Apr 24 23:25:19.127004 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.126990    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.129651 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.129629    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Apr 24 23:25:19.129806 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.129788    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Apr 24 23:25:19.138524 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.138503    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"]
Apr 24 23:25:19.197080 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.197059    2543 generic.go:358] "Generic (PLEG): container finished" podID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerID="3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035" exitCode=2
Apr 24 23:25:19.197178 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.197133    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" event={"ID":"66bbc8ec-2fe9-4d9e-b320-5279848d64ae","Type":"ContainerDied","Data":"3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035"}
Apr 24 23:25:19.205997 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.205977    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40651756-a446-41ae-86da-e7e4c98a1457-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.206078 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.206013    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229fs\" (UniqueName: \"kubernetes.io/projected/40651756-a446-41ae-86da-e7e4c98a1457-kube-api-access-229fs\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.206078 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.206040    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40651756-a446-41ae-86da-e7e4c98a1457-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.206147 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.206090    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40651756-a446-41ae-86da-e7e4c98a1457-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.307138 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.307077    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40651756-a446-41ae-86da-e7e4c98a1457-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.307138 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.307113    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40651756-a446-41ae-86da-e7e4c98a1457-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.307264 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.307147    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40651756-a446-41ae-86da-e7e4c98a1457-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.307264 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.307181    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-229fs\" (UniqueName: \"kubernetes.io/projected/40651756-a446-41ae-86da-e7e4c98a1457-kube-api-access-229fs\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.307533 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.307515    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40651756-a446-41ae-86da-e7e4c98a1457-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.307766 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.307744    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40651756-a446-41ae-86da-e7e4c98a1457-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.309578 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.309557    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40651756-a446-41ae-86da-e7e4c98a1457-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.316322 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.316295    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-229fs\" (UniqueName: \"kubernetes.io/projected/40651756-a446-41ae-86da-e7e4c98a1457-kube-api-access-229fs\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.437480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.437458    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:19.558656 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:19.558580    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"]
Apr 24 23:25:19.561398 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:25:19.561373    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40651756_a446_41ae_86da_e7e4c98a1457.slice/crio-e29ff644538fb1dace5877e588c9ff819711d28cf8df69c49695ddc2da07389c WatchSource:0}: Error finding container e29ff644538fb1dace5877e588c9ff819711d28cf8df69c49695ddc2da07389c: Status 404 returned error can't find the container with id e29ff644538fb1dace5877e588c9ff819711d28cf8df69c49695ddc2da07389c
Apr 24 23:25:20.201749 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:20.201718    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" event={"ID":"40651756-a446-41ae-86da-e7e4c98a1457","Type":"ContainerStarted","Data":"4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e"}
Apr 24 23:25:20.201749 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:20.201753    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" event={"ID":"40651756-a446-41ae-86da-e7e4c98a1457","Type":"ContainerStarted","Data":"e29ff644538fb1dace5877e588c9ff819711d28cf8df69c49695ddc2da07389c"}
Apr 24 23:25:22.020116 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.020076    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Apr 24 23:25:22.024517 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.024492    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Apr 24 23:25:22.139740 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.139707    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:25:22.214596 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.214563    2543 generic.go:358] "Generic (PLEG): container finished" podID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerID="b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa" exitCode=0
Apr 24 23:25:22.214738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.214643    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" event={"ID":"66bbc8ec-2fe9-4d9e-b320-5279848d64ae","Type":"ContainerDied","Data":"b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa"}
Apr 24 23:25:22.214738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.214661    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"
Apr 24 23:25:22.214738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.214677    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb" event={"ID":"66bbc8ec-2fe9-4d9e-b320-5279848d64ae","Type":"ContainerDied","Data":"1b8002724341f9e97d11fd9de4f59b2e5e502d96c6840b093b973b6d35402711"}
Apr 24 23:25:22.214738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.214693    2543 scope.go:117] "RemoveContainer" containerID="3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035"
Apr 24 23:25:22.222353 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.222325    2543 scope.go:117] "RemoveContainer" containerID="b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa"
Apr 24 23:25:22.229121 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.229100    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlrgl\" (UniqueName: \"kubernetes.io/projected/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kube-api-access-rlrgl\") pod \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") "
Apr 24 23:25:22.229312 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.229288    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-proxy-tls\") pod \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") "
Apr 24 23:25:22.229514 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.229331    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") "
Apr 24 23:25:22.229514 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.229428    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kserve-provision-location\") pod \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\" (UID: \"66bbc8ec-2fe9-4d9e-b320-5279848d64ae\") "
Apr 24 23:25:22.229895 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.229856    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-isvc-xgboost-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-runtime-kube-rbac-proxy-sar-config") pod "66bbc8ec-2fe9-4d9e-b320-5279848d64ae" (UID: "66bbc8ec-2fe9-4d9e-b320-5279848d64ae"). InnerVolumeSpecName "isvc-xgboost-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:25:22.229989 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.229898    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66bbc8ec-2fe9-4d9e-b320-5279848d64ae" (UID: "66bbc8ec-2fe9-4d9e-b320-5279848d64ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:25:22.231397 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.231376    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66bbc8ec-2fe9-4d9e-b320-5279848d64ae" (UID: "66bbc8ec-2fe9-4d9e-b320-5279848d64ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:25:22.231473 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.231458    2543 scope.go:117] "RemoveContainer" containerID="a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1"
Apr 24 23:25:22.231814 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.231796    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kube-api-access-rlrgl" (OuterVolumeSpecName: "kube-api-access-rlrgl") pod "66bbc8ec-2fe9-4d9e-b320-5279848d64ae" (UID: "66bbc8ec-2fe9-4d9e-b320-5279848d64ae"). InnerVolumeSpecName "kube-api-access-rlrgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:25:22.243958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.243944    2543 scope.go:117] "RemoveContainer" containerID="3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035"
Apr 24 23:25:22.244204 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:25:22.244186    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035\": container with ID starting with 3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035 not found: ID does not exist" containerID="3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035"
Apr 24 23:25:22.244262 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.244211    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035"} err="failed to get container status \"3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035\": rpc error: code = NotFound desc = could not find container \"3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035\": container with ID starting with 3fde5233c3aa671ef0dfbda9237402eda294aa481152f2b12293864376b7a035 not found: ID does not exist"
Apr 24 23:25:22.244262 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.244229    2543 scope.go:117] "RemoveContainer" containerID="b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa"
Apr 24 23:25:22.244488 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:25:22.244470    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa\": container with ID starting with b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa not found: ID does not exist" containerID="b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa"
Apr 24 23:25:22.244540 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.244493    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa"} err="failed to get container status \"b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa\": rpc error: code = NotFound desc = could not find container \"b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa\": container with ID starting with b218c35c656b402b845b1e31df9ac614bee83e173966545a2c3b549e8cf67faa not found: ID does not exist"
Apr 24 23:25:22.244540 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.244508    2543 scope.go:117] "RemoveContainer" containerID="a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1"
Apr 24 23:25:22.244745 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:25:22.244729    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1\": container with ID starting with a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1 not found: ID does not exist" containerID="a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1"
Apr 24 23:25:22.244790 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.244750    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1"} err="failed to get container status \"a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1\": rpc error: code = NotFound desc = could not find container \"a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1\": container with ID starting with a427ba99374d9aeb98a9113a024ec05aa67e4a1e7b02c8d21530dab5b44f99b1 not found: ID does not exist"
Apr 24 23:25:22.330934 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.330911    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:25:22.330934 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.330931    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlrgl\" (UniqueName: \"kubernetes.io/projected/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-kube-api-access-rlrgl\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:25:22.331063 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.330941    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:25:22.331063 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.330951    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66bbc8ec-2fe9-4d9e-b320-5279848d64ae-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:25:22.535799 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.535772    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"]
Apr 24 23:25:22.539067 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:22.539047    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-9kzfb"]
Apr 24 23:25:23.055485 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:23.055458    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" path="/var/lib/kubelet/pods/66bbc8ec-2fe9-4d9e-b320-5279848d64ae/volumes"
Apr 24 23:25:24.222411 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:24.222381    2543 generic.go:358] "Generic (PLEG): container finished" podID="40651756-a446-41ae-86da-e7e4c98a1457" containerID="4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e" exitCode=0
Apr 24 23:25:24.222772 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:24.222426    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" event={"ID":"40651756-a446-41ae-86da-e7e4c98a1457","Type":"ContainerDied","Data":"4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e"}
Apr 24 23:25:25.227448 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:25.227417    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" event={"ID":"40651756-a446-41ae-86da-e7e4c98a1457","Type":"ContainerStarted","Data":"ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d"}
Apr 24 23:25:25.227448 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:25.227452    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" event={"ID":"40651756-a446-41ae-86da-e7e4c98a1457","Type":"ContainerStarted","Data":"7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a"}
Apr 24 23:25:25.227841 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:25.227658    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:25.227841 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:25.227718    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:25.247309 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:25.247264    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" podStartSLOduration=6.247253655 podStartE2EDuration="6.247253655s" podCreationTimestamp="2026-04-24 23:25:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:25:25.245782091 +0000 UTC m=+3328.672179621" watchObservedRunningTime="2026-04-24 23:25:25.247253655 +0000 UTC m=+3328.673651186"
Apr 24 23:25:31.234767 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:31.234736    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:25:34.934603 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:34.934573    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:25:34.935917 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:25:34.935899    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:26:01.310984 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:01.310937    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Apr 24 23:26:11.237892 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:11.237866    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:26:19.141596 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.141562    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"]
Apr 24 23:26:19.142044 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.141855    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kserve-container" containerID="cri-o://7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a" gracePeriod=30
Apr 24 23:26:19.144369 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.142227    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kube-rbac-proxy" containerID="cri-o://ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d" gracePeriod=30
Apr 24 23:26:19.230455 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.230432    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"]
Apr 24 23:26:19.230741 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.230730    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="storage-initializer"
Apr 24 23:26:19.230741 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.230743    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="storage-initializer"
Apr 24 23:26:19.230825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.230751    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container"
Apr 24 23:26:19.230825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.230757    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container"
Apr 24 23:26:19.230825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.230764    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kube-rbac-proxy"
Apr 24 23:26:19.230825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.230770    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kube-rbac-proxy"
Apr 24 23:26:19.230970 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.230830    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kserve-container"
Apr 24 23:26:19.230970 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.230837    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="66bbc8ec-2fe9-4d9e-b320-5279848d64ae" containerName="kube-rbac-proxy"
Apr 24 23:26:19.235123 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.235108    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.237599 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.237577    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Apr 24 23:26:19.237686 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.237603    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Apr 24 23:26:19.241516 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.241497    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"]
Apr 24 23:26:19.248533 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.248506    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2c9\" (UniqueName: \"kubernetes.io/projected/3d90b47b-0c05-4526-ba18-47d279656ef3-kube-api-access-ht2c9\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.248613 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.248555    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d90b47b-0c05-4526-ba18-47d279656ef3-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.248658 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.248613    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d90b47b-0c05-4526-ba18-47d279656ef3-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.248729 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.248715    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d90b47b-0c05-4526-ba18-47d279656ef3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.349404 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.349376    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d90b47b-0c05-4526-ba18-47d279656ef3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.349510 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.349413    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2c9\" (UniqueName: \"kubernetes.io/projected/3d90b47b-0c05-4526-ba18-47d279656ef3-kube-api-access-ht2c9\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.349510 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.349432    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d90b47b-0c05-4526-ba18-47d279656ef3-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.349510 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.349454    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d90b47b-0c05-4526-ba18-47d279656ef3-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.349673 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:26:19.349537    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Apr 24 23:26:19.349673 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:26:19.349638    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d90b47b-0c05-4526-ba18-47d279656ef3-proxy-tls podName:3d90b47b-0c05-4526-ba18-47d279656ef3 nodeName:}" failed. No retries permitted until 2026-04-24 23:26:19.849622747 +0000 UTC m=+3383.276020256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3d90b47b-0c05-4526-ba18-47d279656ef3-proxy-tls") pod "isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" (UID: "3d90b47b-0c05-4526-ba18-47d279656ef3") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Apr 24 23:26:19.349859 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.349841    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d90b47b-0c05-4526-ba18-47d279656ef3-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.350086 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.350070    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d90b47b-0c05-4526-ba18-47d279656ef3-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.360324 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.360301    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2c9\" (UniqueName: \"kubernetes.io/projected/3d90b47b-0c05-4526-ba18-47d279656ef3-kube-api-access-ht2c9\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.378040 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.378020    2543 generic.go:358] "Generic (PLEG): container finished" podID="40651756-a446-41ae-86da-e7e4c98a1457" containerID="ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d" exitCode=2
Apr 24 23:26:19.378122 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.378086    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" event={"ID":"40651756-a446-41ae-86da-e7e4c98a1457","Type":"ContainerDied","Data":"ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d"}
Apr 24 23:26:19.852039 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.852006    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d90b47b-0c05-4526-ba18-47d279656ef3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:19.854546 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:19.854527    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d90b47b-0c05-4526-ba18-47d279656ef3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-2fv97\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:20.145638 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:20.145545    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:20.265789 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:20.265758    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"]
Apr 24 23:26:20.269369 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:26:20.269326    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d90b47b_0c05_4526_ba18_47d279656ef3.slice/crio-f259e3d386535682bf1aff1633ddcac36c8f228b0f2d31b88ce4a66c785a72cc WatchSource:0}: Error finding container f259e3d386535682bf1aff1633ddcac36c8f228b0f2d31b88ce4a66c785a72cc: Status 404 returned error can't find the container with id f259e3d386535682bf1aff1633ddcac36c8f228b0f2d31b88ce4a66c785a72cc
Apr 24 23:26:20.271175 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:20.271161    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 23:26:20.382033 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:20.381999    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" event={"ID":"3d90b47b-0c05-4526-ba18-47d279656ef3","Type":"ContainerStarted","Data":"a0724588b57ff35c069880c2883664d7092d1a99a94a71c84e978a7871e3e69c"}
Apr 24 23:26:20.382033 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:20.382034    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" event={"ID":"3d90b47b-0c05-4526-ba18-47d279656ef3","Type":"ContainerStarted","Data":"f259e3d386535682bf1aff1633ddcac36c8f228b0f2d31b88ce4a66c785a72cc"}
Apr 24 23:26:21.230533 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:21.230491    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Apr 24 23:26:24.394383 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:24.394294    2543 generic.go:358] "Generic (PLEG): container finished" podID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerID="a0724588b57ff35c069880c2883664d7092d1a99a94a71c84e978a7871e3e69c" exitCode=0
Apr 24 23:26:24.394383 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:24.394366    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" event={"ID":"3d90b47b-0c05-4526-ba18-47d279656ef3","Type":"ContainerDied","Data":"a0724588b57ff35c069880c2883664d7092d1a99a94a71c84e978a7871e3e69c"}
Apr 24 23:26:25.399398 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.399363    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" event={"ID":"3d90b47b-0c05-4526-ba18-47d279656ef3","Type":"ContainerStarted","Data":"8598fa4a1ed83f44c22049fb67497e03aa9a085660cc73f2dbd6d2a8cc288ee8"}
Apr 24 23:26:25.399398 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.399402    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" event={"ID":"3d90b47b-0c05-4526-ba18-47d279656ef3","Type":"ContainerStarted","Data":"1f048c3399b2c0b174cc99da4805494a2ebcbdf1a11969c545b4b10e63c01192"}
Apr 24 23:26:25.399806 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.399623    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:25.418978 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.418933    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podStartSLOduration=6.418919609 podStartE2EDuration="6.418919609s" podCreationTimestamp="2026-04-24 23:26:19 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:26:25.41815459 +0000 UTC m=+3388.844552123" watchObservedRunningTime="2026-04-24 23:26:25.418919609 +0000 UTC m=+3388.845317139"
Apr 24 23:26:25.677868 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.677848    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:26:25.795056 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.795029    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-229fs\" (UniqueName: \"kubernetes.io/projected/40651756-a446-41ae-86da-e7e4c98a1457-kube-api-access-229fs\") pod \"40651756-a446-41ae-86da-e7e4c98a1457\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") "
Apr 24 23:26:25.795056 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.795062    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40651756-a446-41ae-86da-e7e4c98a1457-kserve-provision-location\") pod \"40651756-a446-41ae-86da-e7e4c98a1457\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") "
Apr 24 23:26:25.795300 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.795104    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40651756-a446-41ae-86da-e7e4c98a1457-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"40651756-a446-41ae-86da-e7e4c98a1457\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") "
Apr 24 23:26:25.795300 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.795149    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40651756-a446-41ae-86da-e7e4c98a1457-proxy-tls\") pod \"40651756-a446-41ae-86da-e7e4c98a1457\" (UID: \"40651756-a446-41ae-86da-e7e4c98a1457\") "
Apr 24 23:26:25.795444 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.795396    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40651756-a446-41ae-86da-e7e4c98a1457-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40651756-a446-41ae-86da-e7e4c98a1457" (UID: "40651756-a446-41ae-86da-e7e4c98a1457"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:26:25.795545 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.795523    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40651756-a446-41ae-86da-e7e4c98a1457-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "40651756-a446-41ae-86da-e7e4c98a1457" (UID: "40651756-a446-41ae-86da-e7e4c98a1457"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:26:25.797283 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.797263    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40651756-a446-41ae-86da-e7e4c98a1457-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40651756-a446-41ae-86da-e7e4c98a1457" (UID: "40651756-a446-41ae-86da-e7e4c98a1457"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:26:25.797397 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.797379    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40651756-a446-41ae-86da-e7e4c98a1457-kube-api-access-229fs" (OuterVolumeSpecName: "kube-api-access-229fs") pod "40651756-a446-41ae-86da-e7e4c98a1457" (UID: "40651756-a446-41ae-86da-e7e4c98a1457"). InnerVolumeSpecName "kube-api-access-229fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:26:25.895878 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.895854    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-229fs\" (UniqueName: \"kubernetes.io/projected/40651756-a446-41ae-86da-e7e4c98a1457-kube-api-access-229fs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:26:25.895878 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.895874    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40651756-a446-41ae-86da-e7e4c98a1457-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:26:25.896019 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.895885    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40651756-a446-41ae-86da-e7e4c98a1457-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:26:25.896019 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:25.895897    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40651756-a446-41ae-86da-e7e4c98a1457-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:26:26.403913 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.403871    2543 generic.go:358] "Generic (PLEG): container finished" podID="40651756-a446-41ae-86da-e7e4c98a1457" containerID="7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a" exitCode=0
Apr 24 23:26:26.404312 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.403955    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"
Apr 24 23:26:26.404312 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.403958    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" event={"ID":"40651756-a446-41ae-86da-e7e4c98a1457","Type":"ContainerDied","Data":"7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a"}
Apr 24 23:26:26.404312 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.404000    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl" event={"ID":"40651756-a446-41ae-86da-e7e4c98a1457","Type":"ContainerDied","Data":"e29ff644538fb1dace5877e588c9ff819711d28cf8df69c49695ddc2da07389c"}
Apr 24 23:26:26.404312 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.404020    2543 scope.go:117] "RemoveContainer" containerID="ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d"
Apr 24 23:26:26.404584 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.404556    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:26.405538 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.405489    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 24 23:26:26.412291 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.412209    2543 scope.go:117] "RemoveContainer" containerID="7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a"
Apr 24 23:26:26.419896 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.419881    2543 scope.go:117] "RemoveContainer" containerID="4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e"
Apr 24 23:26:26.426492 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.426471    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"]
Apr 24 23:26:26.426939 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.426922    2543 scope.go:117] "RemoveContainer" containerID="ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d"
Apr 24 23:26:26.427201 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:26:26.427173    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d\": container with ID starting with ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d not found: ID does not exist" containerID="ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d"
Apr 24 23:26:26.427259 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.427204    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d"} err="failed to get container status \"ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d\": rpc error: code = NotFound desc = could not find container \"ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d\": container with ID starting with ded0394b1bdc696f3466a46a6d51ca477e07089a802eadd2a24a83b024e6f78d not found: ID does not exist"
Apr 24 23:26:26.427259 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.427225    2543 scope.go:117] "RemoveContainer" containerID="7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a"
Apr 24 23:26:26.427513 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:26:26.427481    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a\": container with ID starting with 7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a not found: ID does not exist" containerID="7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a"
Apr 24 23:26:26.427618 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.427518    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a"} err="failed to get container status \"7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a\": rpc error: code = NotFound desc = could not find container \"7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a\": container with ID starting with 7fbc936c8f4ad31a7ef04a426e12eb7612457551820ade508600d72d73c7558a not found: ID does not exist"
Apr 24 23:26:26.427618 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.427543    2543 scope.go:117] "RemoveContainer" containerID="4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e"
Apr 24 23:26:26.427892 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:26:26.427869    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e\": container with ID starting with 4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e not found: ID does not exist" containerID="4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e"
Apr 24 23:26:26.427977 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.427900    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e"} err="failed to get container status \"4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e\": rpc error: code = NotFound desc = could not find container \"4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e\": container with ID starting with 4b21cede303590e30f87f9299dd51cf4e806e3c51d3dc8531c143f6e5e2c048e not found: ID does not exist"
Apr 24 23:26:26.429771 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:26.429751    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mb4hl"]
Apr 24 23:26:27.055787 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:27.055760    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40651756-a446-41ae-86da-e7e4c98a1457" path="/var/lib/kubelet/pods/40651756-a446-41ae-86da-e7e4c98a1457/volumes"
Apr 24 23:26:27.408565 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:27.408477    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 24 23:26:32.413229 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:32.413193    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:26:32.413875 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:32.413847    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 24 23:26:42.414320 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:42.414231    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 24 23:26:52.413740 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:26:52.413698    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 24 23:27:02.414628 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:02.414587    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 24 23:27:12.414404 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:12.414328    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 24 23:27:22.414991 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:22.414963    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:27:29.330495 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.330460    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"]
Apr 24 23:27:29.330870 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.330763    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container" containerID="cri-o://1f048c3399b2c0b174cc99da4805494a2ebcbdf1a11969c545b4b10e63c01192" gracePeriod=30
Apr 24 23:27:29.330936 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.330846    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kube-rbac-proxy" containerID="cri-o://8598fa4a1ed83f44c22049fb67497e03aa9a085660cc73f2dbd6d2a8cc288ee8" gracePeriod=30
Apr 24 23:27:29.413689 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.413659    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"]
Apr 24 23:27:29.414008 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.413991    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kube-rbac-proxy"
Apr 24 23:27:29.414083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.414011    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kube-rbac-proxy"
Apr 24 23:27:29.414083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.414025    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kserve-container"
Apr 24 23:27:29.414083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.414033    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kserve-container"
Apr 24 23:27:29.414083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.414047    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="storage-initializer"
Apr 24 23:27:29.414083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.414056    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="storage-initializer"
Apr 24 23:27:29.414367 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.414129    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kube-rbac-proxy"
Apr 24 23:27:29.414367 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.414144    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="40651756-a446-41ae-86da-e7e4c98a1457" containerName="kserve-container"
Apr 24 23:27:29.417174 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.417157    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.419558 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.419534    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Apr 24 23:27:29.419768 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.419752    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Apr 24 23:27:29.419853 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.419822    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Apr 24 23:27:29.424487 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.424468    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"]
Apr 24 23:27:29.466274 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.466253    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcv8b\" (UniqueName: \"kubernetes.io/projected/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kube-api-access-pcv8b\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.466392 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.466289    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.466442 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.466410    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.466489 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.466474    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.567222 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.567196    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pcv8b\" (UniqueName: \"kubernetes.io/projected/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kube-api-access-pcv8b\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.567364 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.567230    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.567364 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.567262    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.567364 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.567292    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.567684 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.567663    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.567912 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.567895    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.569715 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.569696    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.575790 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.575765    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcv8b\" (UniqueName: \"kubernetes.io/projected/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kube-api-access-pcv8b\") pod \"isvc-sklearn-s3-predictor-5bd5d9979-hvskj\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.590194 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.590142    2543 generic.go:358] "Generic (PLEG): container finished" podID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerID="8598fa4a1ed83f44c22049fb67497e03aa9a085660cc73f2dbd6d2a8cc288ee8" exitCode=2
Apr 24 23:27:29.590274 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.590207    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" event={"ID":"3d90b47b-0c05-4526-ba18-47d279656ef3","Type":"ContainerDied","Data":"8598fa4a1ed83f44c22049fb67497e03aa9a085660cc73f2dbd6d2a8cc288ee8"}
Apr 24 23:27:29.727621 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.727592    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:29.845580 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:29.845403    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"]
Apr 24 23:27:29.848080 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:27:29.848057    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f188a6_cb8b_45a2_b8e3_8d25598ee9c2.slice/crio-af64666190e43991a52d89b46b7d9e302adbf51951ff21b9170ca04b1fa0de65 WatchSource:0}: Error finding container af64666190e43991a52d89b46b7d9e302adbf51951ff21b9170ca04b1fa0de65: Status 404 returned error can't find the container with id af64666190e43991a52d89b46b7d9e302adbf51951ff21b9170ca04b1fa0de65
Apr 24 23:27:30.594163 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:30.594125    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" event={"ID":"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2","Type":"ContainerStarted","Data":"566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784"}
Apr 24 23:27:30.594555 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:30.594168    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" event={"ID":"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2","Type":"ContainerStarted","Data":"af64666190e43991a52d89b46b7d9e302adbf51951ff21b9170ca04b1fa0de65"}
Apr 24 23:27:31.598108 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:31.598071    2543 generic.go:358] "Generic (PLEG): container finished" podID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerID="566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784" exitCode=0
Apr 24 23:27:31.598476 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:31.598131    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" event={"ID":"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2","Type":"ContainerDied","Data":"566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784"}
Apr 24 23:27:32.409200 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.409151    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Apr 24 23:27:32.414552 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.414505    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Apr 24 23:27:32.603495 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.603456    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" event={"ID":"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2","Type":"ContainerStarted","Data":"1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da"}
Apr 24 23:27:32.603886 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.603503    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" event={"ID":"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2","Type":"ContainerStarted","Data":"5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295"}
Apr 24 23:27:32.603886 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.603620    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:32.605465 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.605439    2543 generic.go:358] "Generic (PLEG): container finished" podID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerID="1f048c3399b2c0b174cc99da4805494a2ebcbdf1a11969c545b4b10e63c01192" exitCode=0
Apr 24 23:27:32.605604 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.605485    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" event={"ID":"3d90b47b-0c05-4526-ba18-47d279656ef3","Type":"ContainerDied","Data":"1f048c3399b2c0b174cc99da4805494a2ebcbdf1a11969c545b4b10e63c01192"}
Apr 24 23:27:32.627522 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.627477    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podStartSLOduration=3.627455677 podStartE2EDuration="3.627455677s" podCreationTimestamp="2026-04-24 23:27:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:27:32.624765938 +0000 UTC m=+3456.051163468" watchObservedRunningTime="2026-04-24 23:27:32.627455677 +0000 UTC m=+3456.053853185"
Apr 24 23:27:32.670955 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.670936    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:27:32.690083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.690057    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht2c9\" (UniqueName: \"kubernetes.io/projected/3d90b47b-0c05-4526-ba18-47d279656ef3-kube-api-access-ht2c9\") pod \"3d90b47b-0c05-4526-ba18-47d279656ef3\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") "
Apr 24 23:27:32.690190 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.690108    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d90b47b-0c05-4526-ba18-47d279656ef3-kserve-provision-location\") pod \"3d90b47b-0c05-4526-ba18-47d279656ef3\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") "
Apr 24 23:27:32.690190 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.690158    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d90b47b-0c05-4526-ba18-47d279656ef3-proxy-tls\") pod \"3d90b47b-0c05-4526-ba18-47d279656ef3\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") "
Apr 24 23:27:32.690300 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.690194    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d90b47b-0c05-4526-ba18-47d279656ef3-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"3d90b47b-0c05-4526-ba18-47d279656ef3\" (UID: \"3d90b47b-0c05-4526-ba18-47d279656ef3\") "
Apr 24 23:27:32.690694 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.690505    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d90b47b-0c05-4526-ba18-47d279656ef3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d90b47b-0c05-4526-ba18-47d279656ef3" (UID: "3d90b47b-0c05-4526-ba18-47d279656ef3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:27:32.690694 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.690600    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d90b47b-0c05-4526-ba18-47d279656ef3-isvc-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-kube-rbac-proxy-sar-config") pod "3d90b47b-0c05-4526-ba18-47d279656ef3" (UID: "3d90b47b-0c05-4526-ba18-47d279656ef3"). InnerVolumeSpecName "isvc-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:27:32.692236 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.692217    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d90b47b-0c05-4526-ba18-47d279656ef3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d90b47b-0c05-4526-ba18-47d279656ef3" (UID: "3d90b47b-0c05-4526-ba18-47d279656ef3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:27:32.692303 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.692283    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d90b47b-0c05-4526-ba18-47d279656ef3-kube-api-access-ht2c9" (OuterVolumeSpecName: "kube-api-access-ht2c9") pod "3d90b47b-0c05-4526-ba18-47d279656ef3" (UID: "3d90b47b-0c05-4526-ba18-47d279656ef3"). InnerVolumeSpecName "kube-api-access-ht2c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:27:32.791557 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.791531    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2c9\" (UniqueName: \"kubernetes.io/projected/3d90b47b-0c05-4526-ba18-47d279656ef3-kube-api-access-ht2c9\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:27:32.791557 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.791553    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d90b47b-0c05-4526-ba18-47d279656ef3-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:27:32.791707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.791566    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d90b47b-0c05-4526-ba18-47d279656ef3-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:27:32.791707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:32.791575    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d90b47b-0c05-4526-ba18-47d279656ef3-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:27:33.611184 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:33.611137    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97" event={"ID":"3d90b47b-0c05-4526-ba18-47d279656ef3","Type":"ContainerDied","Data":"f259e3d386535682bf1aff1633ddcac36c8f228b0f2d31b88ce4a66c785a72cc"}
Apr 24 23:27:33.611708 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:33.611191    2543 scope.go:117] "RemoveContainer" containerID="8598fa4a1ed83f44c22049fb67497e03aa9a085660cc73f2dbd6d2a8cc288ee8"
Apr 24 23:27:33.611708 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:33.611432    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"
Apr 24 23:27:33.611840 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:33.611728    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:33.612774 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:33.612746    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Apr 24 23:27:33.619172 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:33.619153    2543 scope.go:117] "RemoveContainer" containerID="1f048c3399b2c0b174cc99da4805494a2ebcbdf1a11969c545b4b10e63c01192"
Apr 24 23:27:33.625946 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:33.625927    2543 scope.go:117] "RemoveContainer" containerID="a0724588b57ff35c069880c2883664d7092d1a99a94a71c84e978a7871e3e69c"
Apr 24 23:27:33.628750 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:33.628730    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"]
Apr 24 23:27:33.632500 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:33.632481    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-2fv97"]
Apr 24 23:27:34.616226 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:34.616182    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Apr 24 23:27:35.055576 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:35.055537    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" path="/var/lib/kubelet/pods/3d90b47b-0c05-4526-ba18-47d279656ef3/volumes"
Apr 24 23:27:39.621593 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:39.621566    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:27:39.622090 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:39.622062    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Apr 24 23:27:49.622010 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:49.621968    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Apr 24 23:27:59.623008 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:27:59.622973    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Apr 24 23:28:09.622536 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:09.622454    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Apr 24 23:28:19.622670 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:19.622635    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Apr 24 23:28:29.622639 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:29.622596    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Apr 24 23:28:39.623498 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:39.623466    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:28:49.483984 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.483946    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"]
Apr 24 23:28:49.484548 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.484353    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" containerID="cri-o://5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295" gracePeriod=30
Apr 24 23:28:49.484548 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.484416    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kube-rbac-proxy" containerID="cri-o://1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da" gracePeriod=30
Apr 24 23:28:49.602642 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.602612    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"]
Apr 24 23:28:49.602959 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.602942    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kube-rbac-proxy"
Apr 24 23:28:49.603032 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.602962    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kube-rbac-proxy"
Apr 24 23:28:49.603032 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.602977    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container"
Apr 24 23:28:49.603032 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.602986    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container"
Apr 24 23:28:49.603032 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.603006    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="storage-initializer"
Apr 24 23:28:49.603032 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.603016    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="storage-initializer"
Apr 24 23:28:49.603293 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.603091    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kserve-container"
Apr 24 23:28:49.603293 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.603102    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d90b47b-0c05-4526-ba18-47d279656ef3" containerName="kube-rbac-proxy"
Apr 24 23:28:49.606145 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.606127    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.608512 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.608489    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Apr 24 23:28:49.608591 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.608514    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 24 23:28:49.608643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.608587    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Apr 24 23:28:49.615204 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.615182    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"]
Apr 24 23:28:49.616411 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.616387    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Apr 24 23:28:49.622035 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.622001    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Apr 24 23:28:49.759015 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.758934    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83410060-0085-48af-913d-87dbd73c35f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.759015 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.758973    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83410060-0085-48af-913d-87dbd73c35f8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.759213 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.759024    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62h5k\" (UniqueName: \"kubernetes.io/projected/83410060-0085-48af-913d-87dbd73c35f8-kube-api-access-62h5k\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.759213 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.759054    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.759213 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.759117    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.825648 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.825617    2543 generic.go:358] "Generic (PLEG): container finished" podID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerID="1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da" exitCode=2
Apr 24 23:28:49.825797 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.825694    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" event={"ID":"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2","Type":"ContainerDied","Data":"1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da"}
Apr 24 23:28:49.859969 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.859931    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83410060-0085-48af-913d-87dbd73c35f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.860089 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.859977    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83410060-0085-48af-913d-87dbd73c35f8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.860089 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.860018    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-62h5k\" (UniqueName: \"kubernetes.io/projected/83410060-0085-48af-913d-87dbd73c35f8-kube-api-access-62h5k\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.860089 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.860049    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.860210 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.860087    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.860210 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:28:49.860089    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Apr 24 23:28:49.860304 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:28:49.860223    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83410060-0085-48af-913d-87dbd73c35f8-proxy-tls podName:83410060-0085-48af-913d-87dbd73c35f8 nodeName:}" failed. No retries permitted until 2026-04-24 23:28:50.360203621 +0000 UTC m=+3533.786601129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/83410060-0085-48af-913d-87dbd73c35f8-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" (UID: "83410060-0085-48af-913d-87dbd73c35f8") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Apr 24 23:28:49.860524 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.860497    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83410060-0085-48af-913d-87dbd73c35f8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.860769 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.860752    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.860809 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.860761    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:49.870371 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:49.870322    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62h5k\" (UniqueName: \"kubernetes.io/projected/83410060-0085-48af-913d-87dbd73c35f8-kube-api-access-62h5k\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:50.363375 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:50.363327    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83410060-0085-48af-913d-87dbd73c35f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:50.365818 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:50.365795    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83410060-0085-48af-913d-87dbd73c35f8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:50.516998 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:50.516971    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:50.634943 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:50.634914    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"]
Apr 24 23:28:50.637395 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:28:50.637368    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83410060_0085_48af_913d_87dbd73c35f8.slice/crio-f22404e8a8e2143d085c82838a96135af21aa495c79c9d0c7af418971fe31575 WatchSource:0}: Error finding container f22404e8a8e2143d085c82838a96135af21aa495c79c9d0c7af418971fe31575: Status 404 returned error can't find the container with id f22404e8a8e2143d085c82838a96135af21aa495c79c9d0c7af418971fe31575
Apr 24 23:28:50.830573 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:50.830536    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" event={"ID":"83410060-0085-48af-913d-87dbd73c35f8","Type":"ContainerStarted","Data":"41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383"}
Apr 24 23:28:50.830573 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:50.830574    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" event={"ID":"83410060-0085-48af-913d-87dbd73c35f8","Type":"ContainerStarted","Data":"f22404e8a8e2143d085c82838a96135af21aa495c79c9d0c7af418971fe31575"}
Apr 24 23:28:51.834273 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:51.834240    2543 generic.go:358] "Generic (PLEG): container finished" podID="83410060-0085-48af-913d-87dbd73c35f8" containerID="41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383" exitCode=0
Apr 24 23:28:51.834664 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:51.834302    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" event={"ID":"83410060-0085-48af-913d-87dbd73c35f8","Type":"ContainerDied","Data":"41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383"}
Apr 24 23:28:52.839329 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:52.839290    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" event={"ID":"83410060-0085-48af-913d-87dbd73c35f8","Type":"ContainerStarted","Data":"eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379"}
Apr 24 23:28:52.839329 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:52.839328    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" event={"ID":"83410060-0085-48af-913d-87dbd73c35f8","Type":"ContainerStarted","Data":"2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87"}
Apr 24 23:28:52.839738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:52.839490    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:52.857288 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:52.857246    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podStartSLOduration=3.857234804 podStartE2EDuration="3.857234804s" podCreationTimestamp="2026-04-24 23:28:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:28:52.854898635 +0000 UTC m=+3536.281296166" watchObservedRunningTime="2026-04-24 23:28:52.857234804 +0000 UTC m=+3536.283632337"
Apr 24 23:28:53.528826 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.528804    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:28:53.691914 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.691840    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-proxy-tls\") pod \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") "
Apr 24 23:28:53.691914 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.691886    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kserve-provision-location\") pod \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") "
Apr 24 23:28:53.692114 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.691918    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcv8b\" (UniqueName: \"kubernetes.io/projected/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kube-api-access-pcv8b\") pod \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") "
Apr 24 23:28:53.692114 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.691984    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\" (UID: \"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2\") "
Apr 24 23:28:53.692246 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.692225    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" (UID: "f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:28:53.692289 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.692258    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" (UID: "f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:28:53.694089 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.694066    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" (UID: "f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:28:53.694177 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.694087    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kube-api-access-pcv8b" (OuterVolumeSpecName: "kube-api-access-pcv8b") pod "f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" (UID: "f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2"). InnerVolumeSpecName "kube-api-access-pcv8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:28:53.792756 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.792732    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcv8b\" (UniqueName: \"kubernetes.io/projected/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kube-api-access-pcv8b\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:28:53.792858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.792757    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:28:53.792858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.792771    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:28:53.792858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.792784    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:28:53.843519 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.843489    2543 generic.go:358] "Generic (PLEG): container finished" podID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerID="5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295" exitCode=0
Apr 24 23:28:53.843858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.843572    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"
Apr 24 23:28:53.843858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.843579    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" event={"ID":"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2","Type":"ContainerDied","Data":"5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295"}
Apr 24 23:28:53.843858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.843613    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj" event={"ID":"f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2","Type":"ContainerDied","Data":"af64666190e43991a52d89b46b7d9e302adbf51951ff21b9170ca04b1fa0de65"}
Apr 24 23:28:53.843858 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.843628    2543 scope.go:117] "RemoveContainer" containerID="1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da"
Apr 24 23:28:53.844085 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.844065    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:53.845591 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.845568    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 24 23:28:53.851821 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.851671    2543 scope.go:117] "RemoveContainer" containerID="5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295"
Apr 24 23:28:53.858490 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.858475    2543 scope.go:117] "RemoveContainer" containerID="566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784"
Apr 24 23:28:53.864525 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.864503    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"]
Apr 24 23:28:53.865593 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.865574    2543 scope.go:117] "RemoveContainer" containerID="1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da"
Apr 24 23:28:53.865853 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:28:53.865831    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da\": container with ID starting with 1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da not found: ID does not exist" containerID="1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da"
Apr 24 23:28:53.865930 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.865865    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da"} err="failed to get container status \"1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da\": rpc error: code = NotFound desc = could not find container \"1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da\": container with ID starting with 1c6dcfe0084b1b4145ad0c4e68485dd4a5c340cefea0ec4cfe50525395d851da not found: ID does not exist"
Apr 24 23:28:53.865930 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.865886    2543 scope.go:117] "RemoveContainer" containerID="5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295"
Apr 24 23:28:53.866240 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:28:53.866211    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295\": container with ID starting with 5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295 not found: ID does not exist" containerID="5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295"
Apr 24 23:28:53.866380 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.866249    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295"} err="failed to get container status \"5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295\": rpc error: code = NotFound desc = could not find container \"5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295\": container with ID starting with 5110c76cbd6488c291bb889a66c5a8d43ea0af8da1309b78c9371be033d53295 not found: ID does not exist"
Apr 24 23:28:53.866380 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.866272    2543 scope.go:117] "RemoveContainer" containerID="566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784"
Apr 24 23:28:53.866567 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:28:53.866547    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784\": container with ID starting with 566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784 not found: ID does not exist" containerID="566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784"
Apr 24 23:28:53.866630 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.866572    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784"} err="failed to get container status \"566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784\": rpc error: code = NotFound desc = could not find container \"566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784\": container with ID starting with 566eea6d4859dea811e8b07368f6daac61324bf95ed35ea998b1307af214a784 not found: ID does not exist"
Apr 24 23:28:53.867845 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:53.867825    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5bd5d9979-hvskj"]
Apr 24 23:28:54.847153 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:54.847118    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 24 23:28:55.054869 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:55.054827    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" path="/var/lib/kubelet/pods/f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2/volumes"
Apr 24 23:28:59.852991 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:59.852961    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:28:59.853586 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:28:59.853561    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 24 23:29:09.853528 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:29:09.853491    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 24 23:29:19.853504 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:29:19.853467    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 24 23:29:29.854108 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:29:29.854071    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 24 23:29:39.854225 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:29:39.854144    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 24 23:29:49.854212 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:29:49.854168    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 24 23:29:59.854501 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:29:59.854472    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:30:09.655009 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:09.654976    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"]
Apr 24 23:30:09.655602 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:09.655276    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" containerID="cri-o://2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87" gracePeriod=30
Apr 24 23:30:09.655602 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:09.655329    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kube-rbac-proxy" containerID="cri-o://eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379" gracePeriod=30
Apr 24 23:30:09.847908 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:09.847863    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Apr 24 23:30:09.853474 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:09.853437    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Apr 24 23:30:10.060005 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.059973    2543 generic.go:358] "Generic (PLEG): container finished" podID="83410060-0085-48af-913d-87dbd73c35f8" containerID="eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379" exitCode=2
Apr 24 23:30:10.060139 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.060021    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" event={"ID":"83410060-0085-48af-913d-87dbd73c35f8","Type":"ContainerDied","Data":"eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379"}
Apr 24 23:30:10.746077 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.746045    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"]
Apr 24 23:30:10.746430 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.746327    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="storage-initializer"
Apr 24 23:30:10.746430 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.746352    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="storage-initializer"
Apr 24 23:30:10.746430 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.746368    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kube-rbac-proxy"
Apr 24 23:30:10.746430 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.746374    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kube-rbac-proxy"
Apr 24 23:30:10.746430 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.746384    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container"
Apr 24 23:30:10.746430 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.746390    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container"
Apr 24 23:30:10.746633 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.746436    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kserve-container"
Apr 24 23:30:10.746633 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.746445    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="f5f188a6-cb8b-45a2-b8e3-8d25598ee9c2" containerName="kube-rbac-proxy"
Apr 24 23:30:10.749495 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.749479    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.751725 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.751698    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Apr 24 23:30:10.751725 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.751712    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Apr 24 23:30:10.756964 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.756944    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"]
Apr 24 23:30:10.857781 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.857758    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpj8z\" (UniqueName: \"kubernetes.io/projected/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kube-api-access-lpj8z\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.857890 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.857792    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.857890 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.857837    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.857890 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.857866    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.959179 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.959157    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.959270 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.959190    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.959322 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:30:10.959296    2543 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Apr 24 23:30:10.959398 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.959320    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lpj8z\" (UniqueName: \"kubernetes.io/projected/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kube-api-access-lpj8z\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.959398 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:30:10.959376    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-proxy-tls podName:af9e2c8b-ad15-4b07-b80d-f702cf7df75d nodeName:}" failed. No retries permitted until 2026-04-24 23:30:11.459357044 +0000 UTC m=+3614.885754568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv" (UID: "af9e2c8b-ad15-4b07-b80d-f702cf7df75d") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Apr 24 23:30:10.959506 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.959424    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.959694 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.959676    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.959836 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.959818    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:10.969699 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:10.969671    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpj8z\" (UniqueName: \"kubernetes.io/projected/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kube-api-access-lpj8z\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:11.463378 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:11.463316    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:11.465825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:11.465800    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:11.659743 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:11.659717    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:11.777848 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:11.777818    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"]
Apr 24 23:30:11.781531 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:30:11.781505    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9e2c8b_ad15_4b07_b80d_f702cf7df75d.slice/crio-a22175404dc7da1f7820f5eff43e09d03d2f151538198d0d7d180ca1a4cab1b7 WatchSource:0}: Error finding container a22175404dc7da1f7820f5eff43e09d03d2f151538198d0d7d180ca1a4cab1b7: Status 404 returned error can't find the container with id a22175404dc7da1f7820f5eff43e09d03d2f151538198d0d7d180ca1a4cab1b7
Apr 24 23:30:12.066808 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:12.066733    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv" event={"ID":"af9e2c8b-ad15-4b07-b80d-f702cf7df75d","Type":"ContainerStarted","Data":"eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6"}
Apr 24 23:30:12.066808 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:12.066764    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv" event={"ID":"af9e2c8b-ad15-4b07-b80d-f702cf7df75d","Type":"ContainerStarted","Data":"a22175404dc7da1f7820f5eff43e09d03d2f151538198d0d7d180ca1a4cab1b7"}
Apr 24 23:30:13.495190 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.495167    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:30:13.679407 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.679326    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"83410060-0085-48af-913d-87dbd73c35f8\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") "
Apr 24 23:30:13.679407 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.679383    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83410060-0085-48af-913d-87dbd73c35f8-proxy-tls\") pod \"83410060-0085-48af-913d-87dbd73c35f8\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") "
Apr 24 23:30:13.679605 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.679415    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-cabundle-cert\") pod \"83410060-0085-48af-913d-87dbd73c35f8\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") "
Apr 24 23:30:13.679605 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.679471    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62h5k\" (UniqueName: \"kubernetes.io/projected/83410060-0085-48af-913d-87dbd73c35f8-kube-api-access-62h5k\") pod \"83410060-0085-48af-913d-87dbd73c35f8\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") "
Apr 24 23:30:13.679605 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.679500    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83410060-0085-48af-913d-87dbd73c35f8-kserve-provision-location\") pod \"83410060-0085-48af-913d-87dbd73c35f8\" (UID: \"83410060-0085-48af-913d-87dbd73c35f8\") "
Apr 24 23:30:13.679812 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.679778    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "83410060-0085-48af-913d-87dbd73c35f8" (UID: "83410060-0085-48af-913d-87dbd73c35f8"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:30:13.679919 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.679814    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83410060-0085-48af-913d-87dbd73c35f8" (UID: "83410060-0085-48af-913d-87dbd73c35f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:30:13.679919 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.679851    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83410060-0085-48af-913d-87dbd73c35f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83410060-0085-48af-913d-87dbd73c35f8" (UID: "83410060-0085-48af-913d-87dbd73c35f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:30:13.681765 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.681747    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83410060-0085-48af-913d-87dbd73c35f8-kube-api-access-62h5k" (OuterVolumeSpecName: "kube-api-access-62h5k") pod "83410060-0085-48af-913d-87dbd73c35f8" (UID: "83410060-0085-48af-913d-87dbd73c35f8"). InnerVolumeSpecName "kube-api-access-62h5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:30:13.681850 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.681766    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83410060-0085-48af-913d-87dbd73c35f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83410060-0085-48af-913d-87dbd73c35f8" (UID: "83410060-0085-48af-913d-87dbd73c35f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:30:13.780964 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.780941    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62h5k\" (UniqueName: \"kubernetes.io/projected/83410060-0085-48af-913d-87dbd73c35f8-kube-api-access-62h5k\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:30:13.780964 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.780964    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83410060-0085-48af-913d-87dbd73c35f8-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:30:13.781106 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.780976    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:30:13.781106 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.780985    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83410060-0085-48af-913d-87dbd73c35f8-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:30:13.781106 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:13.780994    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83410060-0085-48af-913d-87dbd73c35f8-cabundle-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:30:14.078497 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.078463    2543 generic.go:358] "Generic (PLEG): container finished" podID="83410060-0085-48af-913d-87dbd73c35f8" containerID="2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87" exitCode=0
Apr 24 23:30:14.078695 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.078546    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" event={"ID":"83410060-0085-48af-913d-87dbd73c35f8","Type":"ContainerDied","Data":"2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87"}
Apr 24 23:30:14.078695 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.078583    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx" event={"ID":"83410060-0085-48af-913d-87dbd73c35f8","Type":"ContainerDied","Data":"f22404e8a8e2143d085c82838a96135af21aa495c79c9d0c7af418971fe31575"}
Apr 24 23:30:14.078695 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.078583    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"
Apr 24 23:30:14.078695 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.078598    2543 scope.go:117] "RemoveContainer" containerID="eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379"
Apr 24 23:30:14.088021 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.088003    2543 scope.go:117] "RemoveContainer" containerID="2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87"
Apr 24 23:30:14.095674 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.095653    2543 scope.go:117] "RemoveContainer" containerID="41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383"
Apr 24 23:30:14.102245 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.102224    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"]
Apr 24 23:30:14.102893 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.102877    2543 scope.go:117] "RemoveContainer" containerID="eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379"
Apr 24 23:30:14.103180 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:30:14.103159    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379\": container with ID starting with eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379 not found: ID does not exist" containerID="eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379"
Apr 24 23:30:14.103252 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.103186    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379"} err="failed to get container status \"eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379\": rpc error: code = NotFound desc = could not find container \"eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379\": container with ID starting with eccaae7b75594077c06e0a0f8d4fc7a14e8c7c973a7560ce8ca92eb509677379 not found: ID does not exist"
Apr 24 23:30:14.103252 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.103202    2543 scope.go:117] "RemoveContainer" containerID="2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87"
Apr 24 23:30:14.103497 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:30:14.103468    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87\": container with ID starting with 2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87 not found: ID does not exist" containerID="2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87"
Apr 24 23:30:14.103600 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.103506    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87"} err="failed to get container status \"2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87\": rpc error: code = NotFound desc = could not find container \"2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87\": container with ID starting with 2d63da4fd9bd2fea69d0b933579f43e15d5d12674440e2f9eae93c374c72ad87 not found: ID does not exist"
Apr 24 23:30:14.103600 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.103531    2543 scope.go:117] "RemoveContainer" containerID="41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383"
Apr 24 23:30:14.103833 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:30:14.103815    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383\": container with ID starting with 41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383 not found: ID does not exist" containerID="41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383"
Apr 24 23:30:14.103884 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.103838    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383"} err="failed to get container status \"41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383\": rpc error: code = NotFound desc = could not find container \"41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383\": container with ID starting with 41bbf236c9f36b4c5f220ae1b69fa4cc3be79acc20008818c62e33bd21ae0383 not found: ID does not exist"
Apr 24 23:30:14.105480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:14.105459    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-786965bbc5-srlbx"]
Apr 24 23:30:15.054825 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:15.054793    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83410060-0085-48af-913d-87dbd73c35f8" path="/var/lib/kubelet/pods/83410060-0085-48af-913d-87dbd73c35f8/volumes"
Apr 24 23:30:16.086530 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:16.086502    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv_af9e2c8b-ad15-4b07-b80d-f702cf7df75d/storage-initializer/0.log"
Apr 24 23:30:16.086876 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:16.086543    2543 generic.go:358] "Generic (PLEG): container finished" podID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" containerID="eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6" exitCode=1
Apr 24 23:30:16.086876 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:16.086597    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv" event={"ID":"af9e2c8b-ad15-4b07-b80d-f702cf7df75d","Type":"ContainerDied","Data":"eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6"}
Apr 24 23:30:17.091494 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:17.091462    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv_af9e2c8b-ad15-4b07-b80d-f702cf7df75d/storage-initializer/0.log"
Apr 24 23:30:17.091862 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:17.091555    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv" event={"ID":"af9e2c8b-ad15-4b07-b80d-f702cf7df75d","Type":"ContainerStarted","Data":"a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e"}
Apr 24 23:30:20.741794 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:20.741762    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"]
Apr 24 23:30:20.742177 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:20.742024    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv" podUID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" containerName="storage-initializer" containerID="cri-o://a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e" gracePeriod=30
Apr 24 23:30:21.287628 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.287608    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv_af9e2c8b-ad15-4b07-b80d-f702cf7df75d/storage-initializer/1.log"
Apr 24 23:30:21.287984 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.287969    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv_af9e2c8b-ad15-4b07-b80d-f702cf7df75d/storage-initializer/0.log"
Apr 24 23:30:21.288050 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.288030    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:21.441974 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.441900    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-proxy-tls\") pod \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") "
Apr 24 23:30:21.442109 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.442005    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kserve-provision-location\") pod \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") "
Apr 24 23:30:21.442109 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.442038    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") "
Apr 24 23:30:21.442109 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.442073    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpj8z\" (UniqueName: \"kubernetes.io/projected/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kube-api-access-lpj8z\") pod \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\" (UID: \"af9e2c8b-ad15-4b07-b80d-f702cf7df75d\") "
Apr 24 23:30:21.442379 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.442327    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af9e2c8b-ad15-4b07-b80d-f702cf7df75d" (UID: "af9e2c8b-ad15-4b07-b80d-f702cf7df75d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:30:21.442462 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.442418    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "af9e2c8b-ad15-4b07-b80d-f702cf7df75d" (UID: "af9e2c8b-ad15-4b07-b80d-f702cf7df75d"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:30:21.444139 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.444111    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af9e2c8b-ad15-4b07-b80d-f702cf7df75d" (UID: "af9e2c8b-ad15-4b07-b80d-f702cf7df75d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:30:21.444233 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.444159    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kube-api-access-lpj8z" (OuterVolumeSpecName: "kube-api-access-lpj8z") pod "af9e2c8b-ad15-4b07-b80d-f702cf7df75d" (UID: "af9e2c8b-ad15-4b07-b80d-f702cf7df75d"). InnerVolumeSpecName "kube-api-access-lpj8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:30:21.543480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.543448    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:30:21.543480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.543478    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpj8z\" (UniqueName: \"kubernetes.io/projected/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kube-api-access-lpj8z\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:30:21.543480 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.543488    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:30:21.543638 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.543498    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af9e2c8b-ad15-4b07-b80d-f702cf7df75d-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:30:21.805616 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805589    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"]
Apr 24 23:30:21.805958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805857    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kube-rbac-proxy"
Apr 24 23:30:21.805958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805868    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kube-rbac-proxy"
Apr 24 23:30:21.805958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805875    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" containerName="storage-initializer"
Apr 24 23:30:21.805958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805880    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" containerName="storage-initializer"
Apr 24 23:30:21.805958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805892    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container"
Apr 24 23:30:21.805958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805897    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container"
Apr 24 23:30:21.805958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805907    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="storage-initializer"
Apr 24 23:30:21.805958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805912    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="storage-initializer"
Apr 24 23:30:21.805958 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805962    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" containerName="storage-initializer"
Apr 24 23:30:21.806293 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805970    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kube-rbac-proxy"
Apr 24 23:30:21.806293 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805976    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="83410060-0085-48af-913d-87dbd73c35f8" containerName="kserve-container"
Apr 24 23:30:21.806293 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.805982    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" containerName="storage-initializer"
Apr 24 23:30:21.806293 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.806032    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" containerName="storage-initializer"
Apr 24 23:30:21.806293 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.806038    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" containerName="storage-initializer"
Apr 24 23:30:21.809234 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.809215    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.811550 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.811530    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Apr 24 23:30:21.811650 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.811559    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Apr 24 23:30:21.811723 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.811705    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 24 23:30:21.819218 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.819193    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"]
Apr 24 23:30:21.846202 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.846181    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b23b09-4a7d-4782-9458-e3ee7f1be038-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.846308 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.846223    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9kd\" (UniqueName: \"kubernetes.io/projected/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kube-api-access-nw9kd\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.846308 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.846245    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.846308 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.846297    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.846451 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.846327    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.947216 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.947194    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b23b09-4a7d-4782-9458-e3ee7f1be038-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.947333 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.947242    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9kd\" (UniqueName: \"kubernetes.io/projected/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kube-api-access-nw9kd\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.947333 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.947266    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.947333 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.947292    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.947333 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.947316    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.947760 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.947726    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.948065 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.948049    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.948111 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.948053    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.949705 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.949688    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b23b09-4a7d-4782-9458-e3ee7f1be038-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:21.954995 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:21.954972    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9kd\" (UniqueName: \"kubernetes.io/projected/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kube-api-access-nw9kd\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:22.107786 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.107729    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv_af9e2c8b-ad15-4b07-b80d-f702cf7df75d/storage-initializer/1.log"
Apr 24 23:30:22.108106 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.108091    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv_af9e2c8b-ad15-4b07-b80d-f702cf7df75d/storage-initializer/0.log"
Apr 24 23:30:22.108156 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.108124    2543 generic.go:358] "Generic (PLEG): container finished" podID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" containerID="a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e" exitCode=1
Apr 24 23:30:22.108195 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.108155    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv" event={"ID":"af9e2c8b-ad15-4b07-b80d-f702cf7df75d","Type":"ContainerDied","Data":"a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e"}
Apr 24 23:30:22.108195 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.108192    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv" event={"ID":"af9e2c8b-ad15-4b07-b80d-f702cf7df75d","Type":"ContainerDied","Data":"a22175404dc7da1f7820f5eff43e09d03d2f151538198d0d7d180ca1a4cab1b7"}
Apr 24 23:30:22.108280 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.108202    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"
Apr 24 23:30:22.108280 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.108208    2543 scope.go:117] "RemoveContainer" containerID="a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e"
Apr 24 23:30:22.117080 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.117012    2543 scope.go:117] "RemoveContainer" containerID="eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6"
Apr 24 23:30:22.121209 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.121189    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:22.124090 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.124074    2543 scope.go:117] "RemoveContainer" containerID="a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e"
Apr 24 23:30:22.124353 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:30:22.124318    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e\": container with ID starting with a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e not found: ID does not exist" containerID="a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e"
Apr 24 23:30:22.124410 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.124366    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e"} err="failed to get container status \"a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e\": rpc error: code = NotFound desc = could not find container \"a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e\": container with ID starting with a4528130b7d8ed9decde1f042a5ee30bd9c2601baf2fb44c8b4d65a62ae3c06e not found: ID does not exist"
Apr 24 23:30:22.124410 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.124384    2543 scope.go:117] "RemoveContainer" containerID="eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6"
Apr 24 23:30:22.124636 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:30:22.124617    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6\": container with ID starting with eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6 not found: ID does not exist" containerID="eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6"
Apr 24 23:30:22.124746 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.124638    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6"} err="failed to get container status \"eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6\": rpc error: code = NotFound desc = could not find container \"eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6\": container with ID starting with eb4c71701deab6a8067d0cae45399a50fd45ec31e128a719071c311f69c84ce6 not found: ID does not exist"
Apr 24 23:30:22.144718 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.144665    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"]
Apr 24 23:30:22.148034 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.148008    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-84c4d8bb85-4kktv"]
Apr 24 23:30:22.251222 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:22.251200    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"]
Apr 24 23:30:22.253704 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:30:22.253678    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b23b09_4a7d_4782_9458_e3ee7f1be038.slice/crio-632d03d6497a268fe14bdabb31331d2f9d245d02529fbc1a2cacaf06fd92d843 WatchSource:0}: Error finding container 632d03d6497a268fe14bdabb31331d2f9d245d02529fbc1a2cacaf06fd92d843: Status 404 returned error can't find the container with id 632d03d6497a268fe14bdabb31331d2f9d245d02529fbc1a2cacaf06fd92d843
Apr 24 23:30:23.056079 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:23.056042    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9e2c8b-ad15-4b07-b80d-f702cf7df75d" path="/var/lib/kubelet/pods/af9e2c8b-ad15-4b07-b80d-f702cf7df75d/volumes"
Apr 24 23:30:23.113114 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:23.113083    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" event={"ID":"f9b23b09-4a7d-4782-9458-e3ee7f1be038","Type":"ContainerStarted","Data":"01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0"}
Apr 24 23:30:23.113114 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:23.113120    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" event={"ID":"f9b23b09-4a7d-4782-9458-e3ee7f1be038","Type":"ContainerStarted","Data":"632d03d6497a268fe14bdabb31331d2f9d245d02529fbc1a2cacaf06fd92d843"}
Apr 24 23:30:24.116908 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:24.116874    2543 generic.go:358] "Generic (PLEG): container finished" podID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerID="01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0" exitCode=0
Apr 24 23:30:24.117256 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:24.116915    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" event={"ID":"f9b23b09-4a7d-4782-9458-e3ee7f1be038","Type":"ContainerDied","Data":"01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0"}
Apr 24 23:30:25.121097 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:25.121054    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" event={"ID":"f9b23b09-4a7d-4782-9458-e3ee7f1be038","Type":"ContainerStarted","Data":"73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943"}
Apr 24 23:30:25.121097 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:25.121103    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" event={"ID":"f9b23b09-4a7d-4782-9458-e3ee7f1be038","Type":"ContainerStarted","Data":"d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb"}
Apr 24 23:30:25.121595 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:25.121264    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:25.121595 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:25.121403    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:25.122670 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:25.122646    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Apr 24 23:30:25.140447 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:25.140409    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podStartSLOduration=4.140398932 podStartE2EDuration="4.140398932s" podCreationTimestamp="2026-04-24 23:30:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:30:25.138118655 +0000 UTC m=+3628.564516185" watchObservedRunningTime="2026-04-24 23:30:25.140398932 +0000 UTC m=+3628.566796499"
Apr 24 23:30:26.124151 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:26.124116    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Apr 24 23:30:31.134517 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:31.134487    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:30:31.135214 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:31.135180    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Apr 24 23:30:34.958114 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:34.958086    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:30:34.960286 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:34.960270    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:30:41.135577 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:41.135534    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Apr 24 23:30:51.135878 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:30:51.135834    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Apr 24 23:31:01.135674 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:01.135631    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Apr 24 23:31:11.135860 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:11.135779    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Apr 24 23:31:21.135243 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:21.135196    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Apr 24 23:31:31.136280 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:31.136253    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:31:31.835054 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:31.835024    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"]
Apr 24 23:31:31.835392 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:31.835329    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container" containerID="cri-o://d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb" gracePeriod=30
Apr 24 23:31:31.835462 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:31.835390    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kube-rbac-proxy" containerID="cri-o://73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943" gracePeriod=30
Apr 24 23:31:32.309928 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:32.309888    2543 generic.go:358] "Generic (PLEG): container finished" podID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerID="73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943" exitCode=2
Apr 24 23:31:32.309928 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:32.309925    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" event={"ID":"f9b23b09-4a7d-4782-9458-e3ee7f1be038","Type":"ContainerDied","Data":"73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943"}
Apr 24 23:31:32.910813 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:32.910784    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"]
Apr 24 23:31:32.913940 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:32.913924    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:32.916218 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:32.916192    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Apr 24 23:31:32.916360 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:32.916235    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Apr 24 23:31:32.922025 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:32.922003    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"]
Apr 24 23:31:33.069357 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.069301    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34ecaeee-be23-427c-977c-6305b29351ba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.069357 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.069357    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34ecaeee-be23-427c-977c-6305b29351ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.069544 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.069411    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89rjp\" (UniqueName: \"kubernetes.io/projected/34ecaeee-be23-427c-977c-6305b29351ba-kube-api-access-89rjp\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.069544 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.069482    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34ecaeee-be23-427c-977c-6305b29351ba-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.170772 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.170707    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34ecaeee-be23-427c-977c-6305b29351ba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.170772 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.170740    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34ecaeee-be23-427c-977c-6305b29351ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.170772 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.170763    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89rjp\" (UniqueName: \"kubernetes.io/projected/34ecaeee-be23-427c-977c-6305b29351ba-kube-api-access-89rjp\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.171012 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.170815    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34ecaeee-be23-427c-977c-6305b29351ba-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.171204 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.171172    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34ecaeee-be23-427c-977c-6305b29351ba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.171460 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.171442    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34ecaeee-be23-427c-977c-6305b29351ba-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.173250 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.173235    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34ecaeee-be23-427c-977c-6305b29351ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.178638 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.178604    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89rjp\" (UniqueName: \"kubernetes.io/projected/34ecaeee-be23-427c-977c-6305b29351ba-kube-api-access-89rjp\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.225226 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.225199    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:33.343441 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.343405    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"]
Apr 24 23:31:33.345643 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:31:33.345616    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ecaeee_be23_427c_977c_6305b29351ba.slice/crio-3cb634f8e2fe67e77e816789fd81ac0966c6d70e87fcd8400ee53f674121b6a5 WatchSource:0}: Error finding container 3cb634f8e2fe67e77e816789fd81ac0966c6d70e87fcd8400ee53f674121b6a5: Status 404 returned error can't find the container with id 3cb634f8e2fe67e77e816789fd81ac0966c6d70e87fcd8400ee53f674121b6a5
Apr 24 23:31:33.347532 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:33.347515    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 24 23:31:34.320702 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:34.320665    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk" event={"ID":"34ecaeee-be23-427c-977c-6305b29351ba","Type":"ContainerStarted","Data":"2b110c868f9c43c852b47c07200829053c0919f98b81c956c137d45f3f6a54f3"}
Apr 24 23:31:34.320702 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:34.320699    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk" event={"ID":"34ecaeee-be23-427c-977c-6305b29351ba","Type":"ContainerStarted","Data":"3cb634f8e2fe67e77e816789fd81ac0966c6d70e87fcd8400ee53f674121b6a5"}
Apr 24 23:31:35.664310 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.664288    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:31:35.792499 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792434    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") "
Apr 24 23:31:35.792499 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792475    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-cabundle-cert\") pod \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") "
Apr 24 23:31:35.792707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792502    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b23b09-4a7d-4782-9458-e3ee7f1be038-proxy-tls\") pod \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") "
Apr 24 23:31:35.792707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792591    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw9kd\" (UniqueName: \"kubernetes.io/projected/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kube-api-access-nw9kd\") pod \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") "
Apr 24 23:31:35.792707 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792668    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kserve-provision-location\") pod \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\" (UID: \"f9b23b09-4a7d-4782-9458-e3ee7f1be038\") "
Apr 24 23:31:35.792850 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792812    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9b23b09-4a7d-4782-9458-e3ee7f1be038" (UID: "f9b23b09-4a7d-4782-9458-e3ee7f1be038"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:31:35.792919 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792841    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "f9b23b09-4a7d-4782-9458-e3ee7f1be038" (UID: "f9b23b09-4a7d-4782-9458-e3ee7f1be038"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:31:35.792989 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792970    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:31:35.793046 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792994    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9b23b09-4a7d-4782-9458-e3ee7f1be038-cabundle-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:31:35.793046 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.792989    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9b23b09-4a7d-4782-9458-e3ee7f1be038" (UID: "f9b23b09-4a7d-4782-9458-e3ee7f1be038"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:31:35.794683 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.794658    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b23b09-4a7d-4782-9458-e3ee7f1be038-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9b23b09-4a7d-4782-9458-e3ee7f1be038" (UID: "f9b23b09-4a7d-4782-9458-e3ee7f1be038"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:31:35.794819 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.794802    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kube-api-access-nw9kd" (OuterVolumeSpecName: "kube-api-access-nw9kd") pod "f9b23b09-4a7d-4782-9458-e3ee7f1be038" (UID: "f9b23b09-4a7d-4782-9458-e3ee7f1be038"). InnerVolumeSpecName "kube-api-access-nw9kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:31:35.893394 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.893367    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:31:35.893490 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.893396    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9b23b09-4a7d-4782-9458-e3ee7f1be038-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:31:35.893490 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:35.893412    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9kd\" (UniqueName: \"kubernetes.io/projected/f9b23b09-4a7d-4782-9458-e3ee7f1be038-kube-api-access-nw9kd\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:31:36.327688 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.327624    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_34ecaeee-be23-427c-977c-6305b29351ba/storage-initializer/0.log"
Apr 24 23:31:36.327688 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.327660    2543 generic.go:358] "Generic (PLEG): container finished" podID="34ecaeee-be23-427c-977c-6305b29351ba" containerID="2b110c868f9c43c852b47c07200829053c0919f98b81c956c137d45f3f6a54f3" exitCode=1
Apr 24 23:31:36.327865 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.327726    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk" event={"ID":"34ecaeee-be23-427c-977c-6305b29351ba","Type":"ContainerDied","Data":"2b110c868f9c43c852b47c07200829053c0919f98b81c956c137d45f3f6a54f3"}
Apr 24 23:31:36.329507 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.329479    2543 generic.go:358] "Generic (PLEG): container finished" podID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerID="d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb" exitCode=0
Apr 24 23:31:36.329605 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.329547    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" event={"ID":"f9b23b09-4a7d-4782-9458-e3ee7f1be038","Type":"ContainerDied","Data":"d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb"}
Apr 24 23:31:36.329605 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.329557    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"
Apr 24 23:31:36.329605 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.329571    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr" event={"ID":"f9b23b09-4a7d-4782-9458-e3ee7f1be038","Type":"ContainerDied","Data":"632d03d6497a268fe14bdabb31331d2f9d245d02529fbc1a2cacaf06fd92d843"}
Apr 24 23:31:36.329605 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.329587    2543 scope.go:117] "RemoveContainer" containerID="73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943"
Apr 24 23:31:36.339210 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.339186    2543 scope.go:117] "RemoveContainer" containerID="d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb"
Apr 24 23:31:36.347022 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.346985    2543 scope.go:117] "RemoveContainer" containerID="01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0"
Apr 24 23:31:36.358991 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.358971    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"]
Apr 24 23:31:36.359895 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.359881    2543 scope.go:117] "RemoveContainer" containerID="73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943"
Apr 24 23:31:36.360116 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:31:36.360100    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943\": container with ID starting with 73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943 not found: ID does not exist" containerID="73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943"
Apr 24 23:31:36.360174 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.360125    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943"} err="failed to get container status \"73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943\": rpc error: code = NotFound desc = could not find container \"73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943\": container with ID starting with 73ed1896df727006c259b40be6e67f1b86f613bd78d0b3ddd38ddc8563d82943 not found: ID does not exist"
Apr 24 23:31:36.360174 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.360142    2543 scope.go:117] "RemoveContainer" containerID="d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb"
Apr 24 23:31:36.360432 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:31:36.360414    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb\": container with ID starting with d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb not found: ID does not exist" containerID="d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb"
Apr 24 23:31:36.360509 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.360436    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb"} err="failed to get container status \"d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb\": rpc error: code = NotFound desc = could not find container \"d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb\": container with ID starting with d0578161109064cdfd5da2489059507778dc6a77ad90a0ad5e3d56378269f2fb not found: ID does not exist"
Apr 24 23:31:36.360509 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.360456    2543 scope.go:117] "RemoveContainer" containerID="01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0"
Apr 24 23:31:36.360681 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:31:36.360666    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0\": container with ID starting with 01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0 not found: ID does not exist" containerID="01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0"
Apr 24 23:31:36.360727 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.360685    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0"} err="failed to get container status \"01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0\": rpc error: code = NotFound desc = could not find container \"01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0\": container with ID starting with 01b0f5b181bded0303b7b61b82f0915439a63aafdb31f09f41130da10fd93eb0 not found: ID does not exist"
Apr 24 23:31:36.363258 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:36.363241    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6c99d9597f-crcgr"]
Apr 24 23:31:37.055610 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:37.055578    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" path="/var/lib/kubelet/pods/f9b23b09-4a7d-4782-9458-e3ee7f1be038/volumes"
Apr 24 23:31:37.335017 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:37.334947    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_34ecaeee-be23-427c-977c-6305b29351ba/storage-initializer/0.log"
Apr 24 23:31:37.335017 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:37.334987    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk" event={"ID":"34ecaeee-be23-427c-977c-6305b29351ba","Type":"ContainerStarted","Data":"96ec8287f0f2be9cb925dc615fb9aad44a40083d797da80ebbe2da976f76fb52"}
Apr 24 23:31:42.349313 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:42.349284    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_34ecaeee-be23-427c-977c-6305b29351ba/storage-initializer/1.log"
Apr 24 23:31:42.349705 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:42.349607    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_34ecaeee-be23-427c-977c-6305b29351ba/storage-initializer/0.log"
Apr 24 23:31:42.349705 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:42.349640    2543 generic.go:358] "Generic (PLEG): container finished" podID="34ecaeee-be23-427c-977c-6305b29351ba" containerID="96ec8287f0f2be9cb925dc615fb9aad44a40083d797da80ebbe2da976f76fb52" exitCode=1
Apr 24 23:31:42.349705 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:42.349673    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk" event={"ID":"34ecaeee-be23-427c-977c-6305b29351ba","Type":"ContainerDied","Data":"96ec8287f0f2be9cb925dc615fb9aad44a40083d797da80ebbe2da976f76fb52"}
Apr 24 23:31:42.349705 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:42.349699    2543 scope.go:117] "RemoveContainer" containerID="2b110c868f9c43c852b47c07200829053c0919f98b81c956c137d45f3f6a54f3"
Apr 24 23:31:42.350086 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:42.350063    2543 scope.go:117] "RemoveContainer" containerID="2b110c868f9c43c852b47c07200829053c0919f98b81c956c137d45f3f6a54f3"
Apr 24 23:31:42.365361 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:31:42.365313    2543 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_kserve-ci-e2e-test_34ecaeee-be23-427c-977c-6305b29351ba_0 in pod sandbox 3cb634f8e2fe67e77e816789fd81ac0966c6d70e87fcd8400ee53f674121b6a5 from index: no such id: '2b110c868f9c43c852b47c07200829053c0919f98b81c956c137d45f3f6a54f3'" containerID="2b110c868f9c43c852b47c07200829053c0919f98b81c956c137d45f3f6a54f3"
Apr 24 23:31:42.365460 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:42.365369    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b110c868f9c43c852b47c07200829053c0919f98b81c956c137d45f3f6a54f3"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_kserve-ci-e2e-test_34ecaeee-be23-427c-977c-6305b29351ba_0 in pod sandbox 3cb634f8e2fe67e77e816789fd81ac0966c6d70e87fcd8400ee53f674121b6a5 from index: no such id: '2b110c868f9c43c852b47c07200829053c0919f98b81c956c137d45f3f6a54f3'"
Apr 24 23:31:42.365584 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:31:42.365551    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_kserve-ci-e2e-test(34ecaeee-be23-427c-977c-6305b29351ba)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk" podUID="34ecaeee-be23-427c-977c-6305b29351ba"
Apr 24 23:31:42.898526 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:42.898490    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"]
Apr 24 23:31:43.353383 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.353356    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_34ecaeee-be23-427c-977c-6305b29351ba/storage-initializer/1.log"
Apr 24 23:31:43.481649 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.481625    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_34ecaeee-be23-427c-977c-6305b29351ba/storage-initializer/1.log"
Apr 24 23:31:43.481754 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.481700    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:43.652256 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.652180    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34ecaeee-be23-427c-977c-6305b29351ba-kserve-provision-location\") pod \"34ecaeee-be23-427c-977c-6305b29351ba\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") "
Apr 24 23:31:43.652256 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.652224    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34ecaeee-be23-427c-977c-6305b29351ba-proxy-tls\") pod \"34ecaeee-be23-427c-977c-6305b29351ba\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") "
Apr 24 23:31:43.652487 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.652324    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34ecaeee-be23-427c-977c-6305b29351ba-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"34ecaeee-be23-427c-977c-6305b29351ba\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") "
Apr 24 23:31:43.652487 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.652382    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89rjp\" (UniqueName: \"kubernetes.io/projected/34ecaeee-be23-427c-977c-6305b29351ba-kube-api-access-89rjp\") pod \"34ecaeee-be23-427c-977c-6305b29351ba\" (UID: \"34ecaeee-be23-427c-977c-6305b29351ba\") "
Apr 24 23:31:43.652487 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.652464    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ecaeee-be23-427c-977c-6305b29351ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34ecaeee-be23-427c-977c-6305b29351ba" (UID: "34ecaeee-be23-427c-977c-6305b29351ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:31:43.652628 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.652599    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34ecaeee-be23-427c-977c-6305b29351ba-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:31:43.652628 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.652607    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ecaeee-be23-427c-977c-6305b29351ba-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "34ecaeee-be23-427c-977c-6305b29351ba" (UID: "34ecaeee-be23-427c-977c-6305b29351ba"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:31:43.654482 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.654452    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ecaeee-be23-427c-977c-6305b29351ba-kube-api-access-89rjp" (OuterVolumeSpecName: "kube-api-access-89rjp") pod "34ecaeee-be23-427c-977c-6305b29351ba" (UID: "34ecaeee-be23-427c-977c-6305b29351ba"). InnerVolumeSpecName "kube-api-access-89rjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:31:43.654482 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.654460    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ecaeee-be23-427c-977c-6305b29351ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34ecaeee-be23-427c-977c-6305b29351ba" (UID: "34ecaeee-be23-427c-977c-6305b29351ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:31:43.753836 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.753804    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34ecaeee-be23-427c-977c-6305b29351ba-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:31:43.753836 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.753836    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34ecaeee-be23-427c-977c-6305b29351ba-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:31:43.753961 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.753847    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89rjp\" (UniqueName: \"kubernetes.io/projected/34ecaeee-be23-427c-977c-6305b29351ba-kube-api-access-89rjp\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:31:43.985186 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985153    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"]
Apr 24 23:31:43.985463 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985450    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ecaeee-be23-427c-977c-6305b29351ba" containerName="storage-initializer"
Apr 24 23:31:43.985509 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985466    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ecaeee-be23-427c-977c-6305b29351ba" containerName="storage-initializer"
Apr 24 23:31:43.985509 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985475    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="storage-initializer"
Apr 24 23:31:43.985509 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985480    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="storage-initializer"
Apr 24 23:31:43.985509 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985490    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ecaeee-be23-427c-977c-6305b29351ba" containerName="storage-initializer"
Apr 24 23:31:43.985509 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985496    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ecaeee-be23-427c-977c-6305b29351ba" containerName="storage-initializer"
Apr 24 23:31:43.985661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985512    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container"
Apr 24 23:31:43.985661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985517    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container"
Apr 24 23:31:43.985661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985524    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kube-rbac-proxy"
Apr 24 23:31:43.985661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985529    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kube-rbac-proxy"
Apr 24 23:31:43.985661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985568    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="34ecaeee-be23-427c-977c-6305b29351ba" containerName="storage-initializer"
Apr 24 23:31:43.985661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985576    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kserve-container"
Apr 24 23:31:43.985661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985585    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="f9b23b09-4a7d-4782-9458-e3ee7f1be038" containerName="kube-rbac-proxy"
Apr 24 23:31:43.985661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.985663    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="34ecaeee-be23-427c-977c-6305b29351ba" containerName="storage-initializer"
Apr 24 23:31:43.989996 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.989980    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:43.992283 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.992263    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Apr 24 23:31:43.992413 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.992309    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Apr 24 23:31:43.992413 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.992309    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Apr 24 23:31:43.997380 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:43.997333    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"]
Apr 24 23:31:44.056409 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.056387    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d869306e-c51e-49c1-8781-cfa2d4d305ab-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.056517 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.056430    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.056517 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.056450    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskcf\" (UniqueName: \"kubernetes.io/projected/d869306e-c51e-49c1-8781-cfa2d4d305ab-kube-api-access-wskcf\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.056517 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.056484    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.056517 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.056502    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d869306e-c51e-49c1-8781-cfa2d4d305ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.157432 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.157407    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d869306e-c51e-49c1-8781-cfa2d4d305ab-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.157539 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.157460    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.157539 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.157484    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wskcf\" (UniqueName: \"kubernetes.io/projected/d869306e-c51e-49c1-8781-cfa2d4d305ab-kube-api-access-wskcf\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.157539 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.157508    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.157539 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.157535    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d869306e-c51e-49c1-8781-cfa2d4d305ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.157895 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.157874    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d869306e-c51e-49c1-8781-cfa2d4d305ab-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.158252 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.158230    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.158424 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.158409    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.160125 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.160104    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d869306e-c51e-49c1-8781-cfa2d4d305ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.165374 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.165328    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskcf\" (UniqueName: \"kubernetes.io/projected/d869306e-c51e-49c1-8781-cfa2d4d305ab-kube-api-access-wskcf\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.301355 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.301262    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:44.358972 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.358946    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk_34ecaeee-be23-427c-977c-6305b29351ba/storage-initializer/1.log"
Apr 24 23:31:44.359390 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.359038    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk" event={"ID":"34ecaeee-be23-427c-977c-6305b29351ba","Type":"ContainerDied","Data":"3cb634f8e2fe67e77e816789fd81ac0966c6d70e87fcd8400ee53f674121b6a5"}
Apr 24 23:31:44.359390 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.359078    2543 scope.go:117] "RemoveContainer" containerID="96ec8287f0f2be9cb925dc615fb9aad44a40083d797da80ebbe2da976f76fb52"
Apr 24 23:31:44.359390 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.359226    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"
Apr 24 23:31:44.393735 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.393699    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"]
Apr 24 23:31:44.398576 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.398552    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6d9bf6b78f-pmwjk"]
Apr 24 23:31:44.422267 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:44.422245    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"]
Apr 24 23:31:44.427980 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:31:44.427953    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd869306e_c51e_49c1_8781_cfa2d4d305ab.slice/crio-dff4736d468b940b4563170ae23f5ecdf05cee61ca87b5c2540c781fa9e8da74 WatchSource:0}: Error finding container dff4736d468b940b4563170ae23f5ecdf05cee61ca87b5c2540c781fa9e8da74: Status 404 returned error can't find the container with id dff4736d468b940b4563170ae23f5ecdf05cee61ca87b5c2540c781fa9e8da74
Apr 24 23:31:45.054912 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:45.054878    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ecaeee-be23-427c-977c-6305b29351ba" path="/var/lib/kubelet/pods/34ecaeee-be23-427c-977c-6305b29351ba/volumes"
Apr 24 23:31:45.364192 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:45.364107    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" event={"ID":"d869306e-c51e-49c1-8781-cfa2d4d305ab","Type":"ContainerStarted","Data":"1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b"}
Apr 24 23:31:45.364192 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:45.364141    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" event={"ID":"d869306e-c51e-49c1-8781-cfa2d4d305ab","Type":"ContainerStarted","Data":"dff4736d468b940b4563170ae23f5ecdf05cee61ca87b5c2540c781fa9e8da74"}
Apr 24 23:31:46.368102 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:46.368064    2543 generic.go:358] "Generic (PLEG): container finished" podID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerID="1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b" exitCode=0
Apr 24 23:31:46.368474 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:46.368150    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" event={"ID":"d869306e-c51e-49c1-8781-cfa2d4d305ab","Type":"ContainerDied","Data":"1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b"}
Apr 24 23:31:47.372667 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:47.372625    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" event={"ID":"d869306e-c51e-49c1-8781-cfa2d4d305ab","Type":"ContainerStarted","Data":"772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a"}
Apr 24 23:31:47.372667 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:47.372674    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" event={"ID":"d869306e-c51e-49c1-8781-cfa2d4d305ab","Type":"ContainerStarted","Data":"10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1"}
Apr 24 23:31:47.373088 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:47.372857    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:47.373088 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:47.373011    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:47.374179 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:47.374156    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Apr 24 23:31:47.392195 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:47.392155    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podStartSLOduration=4.392143816 podStartE2EDuration="4.392143816s" podCreationTimestamp="2026-04-24 23:31:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:31:47.390651671 +0000 UTC m=+3710.817049200" watchObservedRunningTime="2026-04-24 23:31:47.392143816 +0000 UTC m=+3710.818541347"
Apr 24 23:31:48.376694 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:48.376654    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Apr 24 23:31:53.381159 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:53.381132    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:31:53.381656 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:31:53.381631    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Apr 24 23:32:03.382066 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:03.382028    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Apr 24 23:32:13.382534 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:13.382491    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Apr 24 23:32:23.381562 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:23.381525    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Apr 24 23:32:33.382218 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:33.382178    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Apr 24 23:32:43.382596 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:43.382537    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Apr 24 23:32:53.382307 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:53.382281    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:32:54.005904 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:54.005865    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"]
Apr 24 23:32:54.006242 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:54.006195    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container" containerID="cri-o://10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1" gracePeriod=30
Apr 24 23:32:54.006405 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:54.006258    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kube-rbac-proxy" containerID="cri-o://772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a" gracePeriod=30
Apr 24 23:32:54.575430 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:54.575399    2543 generic.go:358] "Generic (PLEG): container finished" podID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerID="772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a" exitCode=2
Apr 24 23:32:54.575786 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:54.575474    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" event={"ID":"d869306e-c51e-49c1-8781-cfa2d4d305ab","Type":"ContainerDied","Data":"772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a"}
Apr 24 23:32:55.077781 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.077751    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"]
Apr 24 23:32:55.081032 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.081015    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.083421 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.083403    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Apr 24 23:32:55.083479 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.083417    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Apr 24 23:32:55.089151 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.089127    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"]
Apr 24 23:32:55.185837 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.185809    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d0928e-3147-4e4c-b8cb-05190a90be95-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.185983 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.185840    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d0928e-3147-4e4c-b8cb-05190a90be95-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.185983 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.185864    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlv5h\" (UniqueName: \"kubernetes.io/projected/40d0928e-3147-4e4c-b8cb-05190a90be95-kube-api-access-dlv5h\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.186107 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.185975    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d0928e-3147-4e4c-b8cb-05190a90be95-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.286823 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.286797    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d0928e-3147-4e4c-b8cb-05190a90be95-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.286957 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.286830    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d0928e-3147-4e4c-b8cb-05190a90be95-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.286957 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.286854    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dlv5h\" (UniqueName: \"kubernetes.io/projected/40d0928e-3147-4e4c-b8cb-05190a90be95-kube-api-access-dlv5h\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.286957 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.286887    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d0928e-3147-4e4c-b8cb-05190a90be95-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.287283 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.287263    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d0928e-3147-4e4c-b8cb-05190a90be95-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.287447 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.287419    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d0928e-3147-4e4c-b8cb-05190a90be95-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.289450 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.289430    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d0928e-3147-4e4c-b8cb-05190a90be95-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.294392 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.294369    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlv5h\" (UniqueName: \"kubernetes.io/projected/40d0928e-3147-4e4c-b8cb-05190a90be95-kube-api-access-dlv5h\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.391908 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.391848    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:32:55.513171 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.513146    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"]
Apr 24 23:32:55.514991 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:32:55.514959    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d0928e_3147_4e4c_b8cb_05190a90be95.slice/crio-c52fbf8232a89ba2627eacb9c46a5ea0d3e3ae88e7ed77e1b3d97906f1af847e WatchSource:0}: Error finding container c52fbf8232a89ba2627eacb9c46a5ea0d3e3ae88e7ed77e1b3d97906f1af847e: Status 404 returned error can't find the container with id c52fbf8232a89ba2627eacb9c46a5ea0d3e3ae88e7ed77e1b3d97906f1af847e
Apr 24 23:32:55.579738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.579714    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn" event={"ID":"40d0928e-3147-4e4c-b8cb-05190a90be95","Type":"ContainerStarted","Data":"5d9d35708f981bd7a10bd7999896f4da65791cab8bcb1824277ee48947aa2726"}
Apr 24 23:32:55.579738 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:55.579746    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn" event={"ID":"40d0928e-3147-4e4c-b8cb-05190a90be95","Type":"ContainerStarted","Data":"c52fbf8232a89ba2627eacb9c46a5ea0d3e3ae88e7ed77e1b3d97906f1af847e"}
Apr 24 23:32:57.755786 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.755767    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:32:57.805413 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.805355    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wskcf\" (UniqueName: \"kubernetes.io/projected/d869306e-c51e-49c1-8781-cfa2d4d305ab-kube-api-access-wskcf\") pod \"d869306e-c51e-49c1-8781-cfa2d4d305ab\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") "
Apr 24 23:32:57.805535 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.805416    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-cabundle-cert\") pod \"d869306e-c51e-49c1-8781-cfa2d4d305ab\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") "
Apr 24 23:32:57.805535 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.805467    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d869306e-c51e-49c1-8781-cfa2d4d305ab-proxy-tls\") pod \"d869306e-c51e-49c1-8781-cfa2d4d305ab\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") "
Apr 24 23:32:57.805535 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.805499    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d869306e-c51e-49c1-8781-cfa2d4d305ab-kserve-provision-location\") pod \"d869306e-c51e-49c1-8781-cfa2d4d305ab\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") "
Apr 24 23:32:57.805535 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.805531    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"d869306e-c51e-49c1-8781-cfa2d4d305ab\" (UID: \"d869306e-c51e-49c1-8781-cfa2d4d305ab\") "
Apr 24 23:32:57.805833 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.805798    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d869306e-c51e-49c1-8781-cfa2d4d305ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d869306e-c51e-49c1-8781-cfa2d4d305ab" (UID: "d869306e-c51e-49c1-8781-cfa2d4d305ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:32:57.805949 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.805867    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d869306e-c51e-49c1-8781-cfa2d4d305ab" (UID: "d869306e-c51e-49c1-8781-cfa2d4d305ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:32:57.805949 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.805906    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "d869306e-c51e-49c1-8781-cfa2d4d305ab" (UID: "d869306e-c51e-49c1-8781-cfa2d4d305ab"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:32:57.807725 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.807705    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d869306e-c51e-49c1-8781-cfa2d4d305ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d869306e-c51e-49c1-8781-cfa2d4d305ab" (UID: "d869306e-c51e-49c1-8781-cfa2d4d305ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:32:57.807807 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.807739    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d869306e-c51e-49c1-8781-cfa2d4d305ab-kube-api-access-wskcf" (OuterVolumeSpecName: "kube-api-access-wskcf") pod "d869306e-c51e-49c1-8781-cfa2d4d305ab" (UID: "d869306e-c51e-49c1-8781-cfa2d4d305ab"). InnerVolumeSpecName "kube-api-access-wskcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:32:57.906451 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.906416    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d869306e-c51e-49c1-8781-cfa2d4d305ab-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:32:57.906451 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.906449    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:32:57.906451 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.906459    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wskcf\" (UniqueName: \"kubernetes.io/projected/d869306e-c51e-49c1-8781-cfa2d4d305ab-kube-api-access-wskcf\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:32:57.906615 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.906470    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d869306e-c51e-49c1-8781-cfa2d4d305ab-cabundle-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:32:57.906615 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:57.906479    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d869306e-c51e-49c1-8781-cfa2d4d305ab-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:32:58.590380 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.590322    2543 generic.go:358] "Generic (PLEG): container finished" podID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerID="10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1" exitCode=0
Apr 24 23:32:58.590535 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.590396    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" event={"ID":"d869306e-c51e-49c1-8781-cfa2d4d305ab","Type":"ContainerDied","Data":"10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1"}
Apr 24 23:32:58.590535 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.590430    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk" event={"ID":"d869306e-c51e-49c1-8781-cfa2d4d305ab","Type":"ContainerDied","Data":"dff4736d468b940b4563170ae23f5ecdf05cee61ca87b5c2540c781fa9e8da74"}
Apr 24 23:32:58.590535 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.590451    2543 scope.go:117] "RemoveContainer" containerID="772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a"
Apr 24 23:32:58.590535 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.590451    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"
Apr 24 23:32:58.598721 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.598704    2543 scope.go:117] "RemoveContainer" containerID="10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1"
Apr 24 23:32:58.605524 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.605498    2543 scope.go:117] "RemoveContainer" containerID="1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b"
Apr 24 23:32:58.611295 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.611272    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"]
Apr 24 23:32:58.612510 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.612497    2543 scope.go:117] "RemoveContainer" containerID="772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a"
Apr 24 23:32:58.612748 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:32:58.612726    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a\": container with ID starting with 772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a not found: ID does not exist" containerID="772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a"
Apr 24 23:32:58.612811 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.612754    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a"} err="failed to get container status \"772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a\": rpc error: code = NotFound desc = could not find container \"772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a\": container with ID starting with 772143ea4ce9ec6160b9bce45893fe7b3f55d841b4e6708608f89325b8e18f7a not found: ID does not exist"
Apr 24 23:32:58.612811 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.612773    2543 scope.go:117] "RemoveContainer" containerID="10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1"
Apr 24 23:32:58.612988 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:32:58.612971    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1\": container with ID starting with 10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1 not found: ID does not exist" containerID="10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1"
Apr 24 23:32:58.613028 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.612995    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1"} err="failed to get container status \"10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1\": rpc error: code = NotFound desc = could not find container \"10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1\": container with ID starting with 10909bb533a906e126436e90590fd9811814a35f28879be23a7e8bbe0968bac1 not found: ID does not exist"
Apr 24 23:32:58.613028 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.613009    2543 scope.go:117] "RemoveContainer" containerID="1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b"
Apr 24 23:32:58.613217 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:32:58.613201    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b\": container with ID starting with 1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b not found: ID does not exist" containerID="1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b"
Apr 24 23:32:58.613279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.613222    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b"} err="failed to get container status \"1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b\": rpc error: code = NotFound desc = could not find container \"1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b\": container with ID starting with 1803f5ee82571887875b4133ee076ee49002f6f975eeaad865845bd9abe40a9b not found: ID does not exist"
Apr 24 23:32:58.615968 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:58.615949    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-69798956bb-k5lvk"]
Apr 24 23:32:59.055619 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:32:59.055589    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" path="/var/lib/kubelet/pods/d869306e-c51e-49c1-8781-cfa2d4d305ab/volumes"
Apr 24 23:33:02.604076 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:02.604050    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_40d0928e-3147-4e4c-b8cb-05190a90be95/storage-initializer/0.log"
Apr 24 23:33:02.604573 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:02.604091    2543 generic.go:358] "Generic (PLEG): container finished" podID="40d0928e-3147-4e4c-b8cb-05190a90be95" containerID="5d9d35708f981bd7a10bd7999896f4da65791cab8bcb1824277ee48947aa2726" exitCode=1
Apr 24 23:33:02.604573 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:02.604135    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn" event={"ID":"40d0928e-3147-4e4c-b8cb-05190a90be95","Type":"ContainerDied","Data":"5d9d35708f981bd7a10bd7999896f4da65791cab8bcb1824277ee48947aa2726"}
Apr 24 23:33:03.609443 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:03.609419    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_40d0928e-3147-4e4c-b8cb-05190a90be95/storage-initializer/0.log"
Apr 24 23:33:03.609836 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:03.609486    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn" event={"ID":"40d0928e-3147-4e4c-b8cb-05190a90be95","Type":"ContainerStarted","Data":"c9b681778e60e3b3ff7897a65848db25667a90392a13c249459b9215d05e0914"}
Apr 24 23:33:04.613551 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:04.613477    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_40d0928e-3147-4e4c-b8cb-05190a90be95/storage-initializer/1.log"
Apr 24 23:33:04.614054 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:04.613829    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_40d0928e-3147-4e4c-b8cb-05190a90be95/storage-initializer/0.log"
Apr 24 23:33:04.614054 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:04.613869    2543 generic.go:358] "Generic (PLEG): container finished" podID="40d0928e-3147-4e4c-b8cb-05190a90be95" containerID="c9b681778e60e3b3ff7897a65848db25667a90392a13c249459b9215d05e0914" exitCode=1
Apr 24 23:33:04.614054 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:04.613944    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn" event={"ID":"40d0928e-3147-4e4c-b8cb-05190a90be95","Type":"ContainerDied","Data":"c9b681778e60e3b3ff7897a65848db25667a90392a13c249459b9215d05e0914"}
Apr 24 23:33:04.614054 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:04.613990    2543 scope.go:117] "RemoveContainer" containerID="5d9d35708f981bd7a10bd7999896f4da65791cab8bcb1824277ee48947aa2726"
Apr 24 23:33:04.614400 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:04.614384    2543 scope.go:117] "RemoveContainer" containerID="5d9d35708f981bd7a10bd7999896f4da65791cab8bcb1824277ee48947aa2726"
Apr 24 23:33:04.625624 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:33:04.625584    2543 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_kserve-ci-e2e-test_40d0928e-3147-4e4c-b8cb-05190a90be95_0 in pod sandbox c52fbf8232a89ba2627eacb9c46a5ea0d3e3ae88e7ed77e1b3d97906f1af847e from index: no such id: '5d9d35708f981bd7a10bd7999896f4da65791cab8bcb1824277ee48947aa2726'" containerID="5d9d35708f981bd7a10bd7999896f4da65791cab8bcb1824277ee48947aa2726"
Apr 24 23:33:04.625682 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:04.625616    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9d35708f981bd7a10bd7999896f4da65791cab8bcb1824277ee48947aa2726"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_kserve-ci-e2e-test_40d0928e-3147-4e4c-b8cb-05190a90be95_0 in pod sandbox c52fbf8232a89ba2627eacb9c46a5ea0d3e3ae88e7ed77e1b3d97906f1af847e from index: no such id: '5d9d35708f981bd7a10bd7999896f4da65791cab8bcb1824277ee48947aa2726'"
Apr 24 23:33:04.626020 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:33:04.626001    2543 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_kserve-ci-e2e-test(40d0928e-3147-4e4c-b8cb-05190a90be95)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn" podUID="40d0928e-3147-4e4c-b8cb-05190a90be95"
Apr 24 23:33:05.286103 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.286072    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"]
Apr 24 23:33:05.618284 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.618212    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_40d0928e-3147-4e4c-b8cb-05190a90be95/storage-initializer/1.log"
Apr 24 23:33:05.738350 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.738303    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_40d0928e-3147-4e4c-b8cb-05190a90be95/storage-initializer/1.log"
Apr 24 23:33:05.738462 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.738409    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:33:05.763251 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.763221    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d0928e-3147-4e4c-b8cb-05190a90be95-kserve-provision-location\") pod \"40d0928e-3147-4e4c-b8cb-05190a90be95\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") "
Apr 24 23:33:05.763379 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.763289    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d0928e-3147-4e4c-b8cb-05190a90be95-proxy-tls\") pod \"40d0928e-3147-4e4c-b8cb-05190a90be95\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") "
Apr 24 23:33:05.763443 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.763374    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d0928e-3147-4e4c-b8cb-05190a90be95-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"40d0928e-3147-4e4c-b8cb-05190a90be95\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") "
Apr 24 23:33:05.763443 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.763401    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlv5h\" (UniqueName: \"kubernetes.io/projected/40d0928e-3147-4e4c-b8cb-05190a90be95-kube-api-access-dlv5h\") pod \"40d0928e-3147-4e4c-b8cb-05190a90be95\" (UID: \"40d0928e-3147-4e4c-b8cb-05190a90be95\") "
Apr 24 23:33:05.763560 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.763504    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d0928e-3147-4e4c-b8cb-05190a90be95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40d0928e-3147-4e4c-b8cb-05190a90be95" (UID: "40d0928e-3147-4e4c-b8cb-05190a90be95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:33:05.763661 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.763640    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40d0928e-3147-4e4c-b8cb-05190a90be95-kserve-provision-location\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:33:05.763786 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.763764    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d0928e-3147-4e4c-b8cb-05190a90be95-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "40d0928e-3147-4e4c-b8cb-05190a90be95" (UID: "40d0928e-3147-4e4c-b8cb-05190a90be95"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 24 23:33:05.765481 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.765456    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d0928e-3147-4e4c-b8cb-05190a90be95-kube-api-access-dlv5h" (OuterVolumeSpecName: "kube-api-access-dlv5h") pod "40d0928e-3147-4e4c-b8cb-05190a90be95" (UID: "40d0928e-3147-4e4c-b8cb-05190a90be95"). InnerVolumeSpecName "kube-api-access-dlv5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:33:05.765577 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.765485    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d0928e-3147-4e4c-b8cb-05190a90be95-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40d0928e-3147-4e4c-b8cb-05190a90be95" (UID: "40d0928e-3147-4e4c-b8cb-05190a90be95"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 24 23:33:05.864613 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.864594    2543 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40d0928e-3147-4e4c-b8cb-05190a90be95-proxy-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:33:05.864613 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.864613    2543 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40d0928e-3147-4e4c-b8cb-05190a90be95-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:33:05.864739 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:05.864623    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlv5h\" (UniqueName: \"kubernetes.io/projected/40d0928e-3147-4e4c-b8cb-05190a90be95-kube-api-access-dlv5h\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:33:06.622017 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:06.621985    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn_40d0928e-3147-4e4c-b8cb-05190a90be95/storage-initializer/1.log"
Apr 24 23:33:06.622428 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:06.622076    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn" event={"ID":"40d0928e-3147-4e4c-b8cb-05190a90be95","Type":"ContainerDied","Data":"c52fbf8232a89ba2627eacb9c46a5ea0d3e3ae88e7ed77e1b3d97906f1af847e"}
Apr 24 23:33:06.622428 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:06.622100    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"
Apr 24 23:33:06.622428 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:06.622109    2543 scope.go:117] "RemoveContainer" containerID="c9b681778e60e3b3ff7897a65848db25667a90392a13c249459b9215d05e0914"
Apr 24 23:33:06.655043 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:06.655015    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"]
Apr 24 23:33:06.658781 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:06.658761    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6cb95fcd5b-8qwqn"]
Apr 24 23:33:07.055219 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.055192    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d0928e-3147-4e4c-b8cb-05190a90be95" path="/var/lib/kubelet/pods/40d0928e-3147-4e4c-b8cb-05190a90be95/volumes"
Apr 24 23:33:07.388096 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388026    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-444hf/must-gather-qv287"]
Apr 24 23:33:07.388296 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388285    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="storage-initializer"
Apr 24 23:33:07.388361 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388299    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="storage-initializer"
Apr 24 23:33:07.388361 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388311    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kube-rbac-proxy"
Apr 24 23:33:07.388361 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388316    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kube-rbac-proxy"
Apr 24 23:33:07.388361 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388324    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d0928e-3147-4e4c-b8cb-05190a90be95" containerName="storage-initializer"
Apr 24 23:33:07.388361 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388329    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d0928e-3147-4e4c-b8cb-05190a90be95" containerName="storage-initializer"
Apr 24 23:33:07.388361 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388355    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d0928e-3147-4e4c-b8cb-05190a90be95" containerName="storage-initializer"
Apr 24 23:33:07.388361 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388360    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d0928e-3147-4e4c-b8cb-05190a90be95" containerName="storage-initializer"
Apr 24 23:33:07.388590 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388372    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container"
Apr 24 23:33:07.388590 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388377    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container"
Apr 24 23:33:07.388590 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388420    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kserve-container"
Apr 24 23:33:07.388590 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388428    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="d869306e-c51e-49c1-8781-cfa2d4d305ab" containerName="kube-rbac-proxy"
Apr 24 23:33:07.388590 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388435    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="40d0928e-3147-4e4c-b8cb-05190a90be95" containerName="storage-initializer"
Apr 24 23:33:07.388590 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.388516    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="40d0928e-3147-4e4c-b8cb-05190a90be95" containerName="storage-initializer"
Apr 24 23:33:07.392917 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.392899    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:07.395415 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.395392    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-444hf\"/\"openshift-service-ca.crt\""
Apr 24 23:33:07.395525 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.395392    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-444hf\"/\"kube-root-ca.crt\""
Apr 24 23:33:07.399994 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.399971    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-444hf/must-gather-qv287"]
Apr 24 23:33:07.477527 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.477501    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-must-gather-output\") pod \"must-gather-qv287\" (UID: \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\") " pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:07.477669 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.477549    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6m6x\" (UniqueName: \"kubernetes.io/projected/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-kube-api-access-k6m6x\") pod \"must-gather-qv287\" (UID: \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\") " pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:07.578743 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.578704    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k6m6x\" (UniqueName: \"kubernetes.io/projected/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-kube-api-access-k6m6x\") pod \"must-gather-qv287\" (UID: \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\") " pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:07.578860 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.578791    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-must-gather-output\") pod \"must-gather-qv287\" (UID: \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\") " pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:07.579083 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.579063    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-must-gather-output\") pod \"must-gather-qv287\" (UID: \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\") " pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:07.586072 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.586051    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6m6x\" (UniqueName: \"kubernetes.io/projected/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-kube-api-access-k6m6x\") pod \"must-gather-qv287\" (UID: \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\") " pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:07.715612 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.715584    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:07.833717 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:07.833689    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-444hf/must-gather-qv287"]
Apr 24 23:33:07.837370 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:33:07.837324    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadbd48b1_8d70_41b8_a0dc_6fa77d8033b7.slice/crio-419d2f326e1158fd99a20a8625b7165cdfe208a4582b21af2ae99650d05c766d WatchSource:0}: Error finding container 419d2f326e1158fd99a20a8625b7165cdfe208a4582b21af2ae99650d05c766d: Status 404 returned error can't find the container with id 419d2f326e1158fd99a20a8625b7165cdfe208a4582b21af2ae99650d05c766d
Apr 24 23:33:08.631668 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:08.631605    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-444hf/must-gather-qv287" event={"ID":"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7","Type":"ContainerStarted","Data":"419d2f326e1158fd99a20a8625b7165cdfe208a4582b21af2ae99650d05c766d"}
Apr 24 23:33:12.646691 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:12.646648    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-444hf/must-gather-qv287" event={"ID":"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7","Type":"ContainerStarted","Data":"6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321"}
Apr 24 23:33:12.647069 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:12.646699    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-444hf/must-gather-qv287" event={"ID":"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7","Type":"ContainerStarted","Data":"1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8"}
Apr 24 23:33:12.664457 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:12.664008    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-444hf/must-gather-qv287" podStartSLOduration=1.4937463819999999 podStartE2EDuration="5.66399006s" podCreationTimestamp="2026-04-24 23:33:07 +0000 UTC" firstStartedPulling="2026-04-24 23:33:07.838850417 +0000 UTC m=+3791.265247925" lastFinishedPulling="2026-04-24 23:33:12.009094092 +0000 UTC m=+3795.435491603" observedRunningTime="2026-04-24 23:33:12.662086703 +0000 UTC m=+3796.088484233" watchObservedRunningTime="2026-04-24 23:33:12.66399006 +0000 UTC m=+3796.090387592"
Apr 24 23:33:32.713993 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:32.713959    2543 generic.go:358] "Generic (PLEG): container finished" podID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" containerID="1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8" exitCode=0
Apr 24 23:33:32.714403 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:32.714011    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-444hf/must-gather-qv287" event={"ID":"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7","Type":"ContainerDied","Data":"1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8"}
Apr 24 23:33:32.714403 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:32.714325    2543 scope.go:117] "RemoveContainer" containerID="1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8"
Apr 24 23:33:33.621259 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:33.621233    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-444hf_must-gather-qv287_adbd48b1-8d70-41b8-a0dc-6fa77d8033b7/gather/0.log"
Apr 24 23:33:34.244205 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.244172    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rmwdq/must-gather-qqnkv"]
Apr 24 23:33:34.246251 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.246234    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmwdq/must-gather-qqnkv"
Apr 24 23:33:34.248643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.248622    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rmwdq\"/\"openshift-service-ca.crt\""
Apr 24 23:33:34.249650 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.249636    2543 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rmwdq\"/\"default-dockercfg-z2mgt\""
Apr 24 23:33:34.249710 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.249646    2543 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rmwdq\"/\"kube-root-ca.crt\""
Apr 24 23:33:34.253400 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.253382    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmwdq/must-gather-qqnkv"]
Apr 24 23:33:34.409708 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.409683    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7dzh\" (UniqueName: \"kubernetes.io/projected/a877d418-c7b4-4c1a-96d4-60c8f72ea709-kube-api-access-s7dzh\") pod \"must-gather-qqnkv\" (UID: \"a877d418-c7b4-4c1a-96d4-60c8f72ea709\") " pod="openshift-must-gather-rmwdq/must-gather-qqnkv"
Apr 24 23:33:34.409824 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.409718    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a877d418-c7b4-4c1a-96d4-60c8f72ea709-must-gather-output\") pod \"must-gather-qqnkv\" (UID: \"a877d418-c7b4-4c1a-96d4-60c8f72ea709\") " pod="openshift-must-gather-rmwdq/must-gather-qqnkv"
Apr 24 23:33:34.510827 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.510772    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s7dzh\" (UniqueName: \"kubernetes.io/projected/a877d418-c7b4-4c1a-96d4-60c8f72ea709-kube-api-access-s7dzh\") pod \"must-gather-qqnkv\" (UID: \"a877d418-c7b4-4c1a-96d4-60c8f72ea709\") " pod="openshift-must-gather-rmwdq/must-gather-qqnkv"
Apr 24 23:33:34.510827 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.510809    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a877d418-c7b4-4c1a-96d4-60c8f72ea709-must-gather-output\") pod \"must-gather-qqnkv\" (UID: \"a877d418-c7b4-4c1a-96d4-60c8f72ea709\") " pod="openshift-must-gather-rmwdq/must-gather-qqnkv"
Apr 24 23:33:34.511103 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.511089    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a877d418-c7b4-4c1a-96d4-60c8f72ea709-must-gather-output\") pod \"must-gather-qqnkv\" (UID: \"a877d418-c7b4-4c1a-96d4-60c8f72ea709\") " pod="openshift-must-gather-rmwdq/must-gather-qqnkv"
Apr 24 23:33:34.519094 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.519069    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7dzh\" (UniqueName: \"kubernetes.io/projected/a877d418-c7b4-4c1a-96d4-60c8f72ea709-kube-api-access-s7dzh\") pod \"must-gather-qqnkv\" (UID: \"a877d418-c7b4-4c1a-96d4-60c8f72ea709\") " pod="openshift-must-gather-rmwdq/must-gather-qqnkv"
Apr 24 23:33:34.555075 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.555057    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmwdq/must-gather-qqnkv"
Apr 24 23:33:34.673251 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.673232    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmwdq/must-gather-qqnkv"]
Apr 24 23:33:34.678964 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:33:34.675937    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda877d418_c7b4_4c1a_96d4_60c8f72ea709.slice/crio-2770d8b0ad7d51a39975fdf980e3900fe70dc96a51beee364d3b24375e241411 WatchSource:0}: Error finding container 2770d8b0ad7d51a39975fdf980e3900fe70dc96a51beee364d3b24375e241411: Status 404 returned error can't find the container with id 2770d8b0ad7d51a39975fdf980e3900fe70dc96a51beee364d3b24375e241411
Apr 24 23:33:34.720665 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:34.720637    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmwdq/must-gather-qqnkv" event={"ID":"a877d418-c7b4-4c1a-96d4-60c8f72ea709","Type":"ContainerStarted","Data":"2770d8b0ad7d51a39975fdf980e3900fe70dc96a51beee364d3b24375e241411"}
Apr 24 23:33:35.725796 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:35.725733    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmwdq/must-gather-qqnkv" event={"ID":"a877d418-c7b4-4c1a-96d4-60c8f72ea709","Type":"ContainerStarted","Data":"8f80595dffd5477c254db520c397d0282a4aa2a8381b6865e6588f469d258758"}
Apr 24 23:33:36.731788 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:36.731051    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmwdq/must-gather-qqnkv" event={"ID":"a877d418-c7b4-4c1a-96d4-60c8f72ea709","Type":"ContainerStarted","Data":"c9a726fbd75e754a44824db831dde8ddfc579be363b4fa67e759f0f132c0c4c1"}
Apr 24 23:33:36.748132 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:36.748062    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-rmwdq/must-gather-qqnkv" podStartSLOduration=1.910483862 podStartE2EDuration="2.748048561s" podCreationTimestamp="2026-04-24 23:33:34 +0000 UTC" firstStartedPulling="2026-04-24 23:33:34.680070641 +0000 UTC m=+3818.106468167" lastFinishedPulling="2026-04-24 23:33:35.517635354 +0000 UTC m=+3818.944032866" observedRunningTime="2026-04-24 23:33:36.746605344 +0000 UTC m=+3820.173002875" watchObservedRunningTime="2026-04-24 23:33:36.748048561 +0000 UTC m=+3820.174446090"
Apr 24 23:33:37.093040 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:37.092959    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kcq9z_d17a86f7-36e1-4352-92cf-748b8c466f7e/global-pull-secret-syncer/0.log"
Apr 24 23:33:37.310995 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:37.310965    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zkmd6_e170e736-789d-406e-a238-bcaa098a2309/konnectivity-agent/0.log"
Apr 24 23:33:37.333880 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:37.333831    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-116.ec2.internal_60d03c02f9ed5519c8c0179698b80bb4/haproxy/0.log"
Apr 24 23:33:39.087852 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.087808    2543 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-444hf/must-gather-qv287"]
Apr 24 23:33:39.088492 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.088116    2543 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-444hf/must-gather-qv287" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" containerName="copy" containerID="cri-o://6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321" gracePeriod=2
Apr 24 23:33:39.091389 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.091363    2543 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-444hf/must-gather-qv287"]
Apr 24 23:33:39.091836 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.091809    2543 status_manager.go:895] "Failed to get status for pod" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" pod="openshift-must-gather-444hf/must-gather-qv287" err="pods \"must-gather-qv287\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-444hf\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object"
Apr 24 23:33:39.438227 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.438147    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-444hf_must-gather-qv287_adbd48b1-8d70-41b8-a0dc-6fa77d8033b7/copy/0.log"
Apr 24 23:33:39.439374 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.438628    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:39.440652 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.440608    2543 status_manager.go:895] "Failed to get status for pod" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" pod="openshift-must-gather-444hf/must-gather-qv287" err="pods \"must-gather-qv287\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-444hf\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object"
Apr 24 23:33:39.566372 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.566314    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6m6x\" (UniqueName: \"kubernetes.io/projected/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-kube-api-access-k6m6x\") pod \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\" (UID: \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\") "
Apr 24 23:33:39.566664 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.566394    2543 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-must-gather-output\") pod \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\" (UID: \"adbd48b1-8d70-41b8-a0dc-6fa77d8033b7\") "
Apr 24 23:33:39.568015 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.567980    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" (UID: "adbd48b1-8d70-41b8-a0dc-6fa77d8033b7"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 24 23:33:39.576861 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.576528    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-kube-api-access-k6m6x" (OuterVolumeSpecName: "kube-api-access-k6m6x") pod "adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" (UID: "adbd48b1-8d70-41b8-a0dc-6fa77d8033b7"). InnerVolumeSpecName "kube-api-access-k6m6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 24 23:33:39.667460 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.667310    2543 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-must-gather-output\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:33:39.667460 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.667393    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6m6x\" (UniqueName: \"kubernetes.io/projected/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7-kube-api-access-k6m6x\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Apr 24 23:33:39.743538 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.743462    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-444hf_must-gather-qv287_adbd48b1-8d70-41b8-a0dc-6fa77d8033b7/copy/0.log"
Apr 24 23:33:39.745375 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.744052    2543 generic.go:358] "Generic (PLEG): container finished" podID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" containerID="6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321" exitCode=143
Apr 24 23:33:39.745375 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.744176    2543 scope.go:117] "RemoveContainer" containerID="6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321"
Apr 24 23:33:39.745375 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.744302    2543 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-444hf/must-gather-qv287"
Apr 24 23:33:39.753670 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.753596    2543 status_manager.go:895] "Failed to get status for pod" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" pod="openshift-must-gather-444hf/must-gather-qv287" err="pods \"must-gather-qv287\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-444hf\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object"
Apr 24 23:33:39.763582 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.763553    2543 status_manager.go:895] "Failed to get status for pod" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" pod="openshift-must-gather-444hf/must-gather-qv287" err="pods \"must-gather-qv287\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-444hf\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object"
Apr 24 23:33:39.765642 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.765617    2543 scope.go:117] "RemoveContainer" containerID="1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8"
Apr 24 23:33:39.792103 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.792064    2543 scope.go:117] "RemoveContainer" containerID="6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321"
Apr 24 23:33:39.792638 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:33:39.792616    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321\": container with ID starting with 6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321 not found: ID does not exist" containerID="6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321"
Apr 24 23:33:39.792818 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.792791    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321"} err="failed to get container status \"6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321\": rpc error: code = NotFound desc = could not find container \"6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321\": container with ID starting with 6d10cba6663feffff1e803036e7efb5727a1cd0cffe663c29f1a88762aa09321 not found: ID does not exist"
Apr 24 23:33:39.792956 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.792943    2543 scope.go:117] "RemoveContainer" containerID="1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8"
Apr 24 23:33:39.793422 ip-10-0-134-116 kubenswrapper[2543]: E0424 23:33:39.793389    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8\": container with ID starting with 1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8 not found: ID does not exist" containerID="1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8"
Apr 24 23:33:39.793571 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:39.793551    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8"} err="failed to get container status \"1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8\": rpc error: code = NotFound desc = could not find container \"1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8\": container with ID starting with 1003dce3d9bb6e7f8e0a961a51df877d4b74c3ed116b15c6d51b794f717089a8 not found: ID does not exist"
Apr 24 23:33:40.672328 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:40.672294    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9d6868c76-bqsqz_3d9e8dd3-d002-4d9d-a0b9-3f3757b1fb5b/metrics-server/0.log"
Apr 24 23:33:40.696045 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:40.696013    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7dccd58f55-w2qcz_d939fa26-4fe1-4af1-913c-cd1f44f0d81d/monitoring-plugin/0.log"
Apr 24 23:33:40.804506 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:40.804471    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qth7q_762ece34-73cf-4ee7-b126-0c07a51efe69/node-exporter/0.log"
Apr 24 23:33:40.826554 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:40.826523    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qth7q_762ece34-73cf-4ee7-b126-0c07a51efe69/kube-rbac-proxy/0.log"
Apr 24 23:33:40.850319 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:40.850228    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qth7q_762ece34-73cf-4ee7-b126-0c07a51efe69/init-textfile/0.log"
Apr 24 23:33:41.051158 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.051126    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_876ba290-f613-46a3-a6b5-a1071a85498f/prometheus/0.log"
Apr 24 23:33:41.055830 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.055798    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" path="/var/lib/kubelet/pods/adbd48b1-8d70-41b8-a0dc-6fa77d8033b7/volumes"
Apr 24 23:33:41.066061 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.066036    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_876ba290-f613-46a3-a6b5-a1071a85498f/config-reloader/0.log"
Apr 24 23:33:41.087635 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.087577    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_876ba290-f613-46a3-a6b5-a1071a85498f/thanos-sidecar/0.log"
Apr 24 23:33:41.108255 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.108186    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_876ba290-f613-46a3-a6b5-a1071a85498f/kube-rbac-proxy-web/0.log"
Apr 24 23:33:41.129283 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.129255    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_876ba290-f613-46a3-a6b5-a1071a85498f/kube-rbac-proxy/0.log"
Apr 24 23:33:41.150430 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.150402    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_876ba290-f613-46a3-a6b5-a1071a85498f/kube-rbac-proxy-thanos/0.log"
Apr 24 23:33:41.172264 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.172234    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_876ba290-f613-46a3-a6b5-a1071a85498f/init-config-reloader/0.log"
Apr 24 23:33:41.249009 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.248819    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-57cf98b594-h9j56_7129347e-47f7-4b88-8fb1-7fbba551e7b3/prometheus-operator-admission-webhook/0.log"
Apr 24 23:33:41.375120 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.375012    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8d8c58-rzj8p_4767d757-18ad-4679-9809-9eb8d54fd794/thanos-query/0.log"
Apr 24 23:33:41.396906 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.396868    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8d8c58-rzj8p_4767d757-18ad-4679-9809-9eb8d54fd794/kube-rbac-proxy-web/0.log"
Apr 24 23:33:41.417845 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.417807    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8d8c58-rzj8p_4767d757-18ad-4679-9809-9eb8d54fd794/kube-rbac-proxy/0.log"
Apr 24 23:33:41.437318 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.437295    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8d8c58-rzj8p_4767d757-18ad-4679-9809-9eb8d54fd794/prom-label-proxy/0.log"
Apr 24 23:33:41.462729 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.462704    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8d8c58-rzj8p_4767d757-18ad-4679-9809-9eb8d54fd794/kube-rbac-proxy-rules/0.log"
Apr 24 23:33:41.486041 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:41.486007    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d5d8d8c58-rzj8p_4767d757-18ad-4679-9809-9eb8d54fd794/kube-rbac-proxy-metrics/0.log"
Apr 24 23:33:43.551269 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:43.551219    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6bcc868b7-q5ch2_034c86c9-fa44-48c0-b7aa-b0f441d9785e/download-server/0.log"
Apr 24 23:33:44.104165 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.104113    2543 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"]
Apr 24 23:33:44.104836 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.104811    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" containerName="gather"
Apr 24 23:33:44.104977 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.104964    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" containerName="gather"
Apr 24 23:33:44.105087 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.105075    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" containerName="copy"
Apr 24 23:33:44.105173 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.105163    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" containerName="copy"
Apr 24 23:33:44.105366 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.105331    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" containerName="gather"
Apr 24 23:33:44.105492 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.105479    2543 memory_manager.go:356] "RemoveStaleState removing state" podUID="adbd48b1-8d70-41b8-a0dc-6fa77d8033b7" containerName="copy"
Apr 24 23:33:44.110133 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.110106    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.115925 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.115901    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"]
Apr 24 23:33:44.209702 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.209430    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-sys\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.209702 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.209472    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-proc\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.209702 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.209520    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-lib-modules\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.209702 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.209579    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7gcr\" (UniqueName: \"kubernetes.io/projected/4622b91e-13d5-42f7-b636-0432434e62f4-kube-api-access-h7gcr\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.209702 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.209615    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-podres\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.311493 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.310647    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h7gcr\" (UniqueName: \"kubernetes.io/projected/4622b91e-13d5-42f7-b636-0432434e62f4-kube-api-access-h7gcr\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.311493 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.310701    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-podres\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.311493 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.310773    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-sys\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.311493 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.310798    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-proc\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.311493 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.310841    2543 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-lib-modules\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.311493 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.311012    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-lib-modules\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.311952 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.311521    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-podres\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.311952 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.311528    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-sys\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.311952 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.311558    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4622b91e-13d5-42f7-b636-0432434e62f4-proc\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.319090 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.319059    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7gcr\" (UniqueName: \"kubernetes.io/projected/4622b91e-13d5-42f7-b636-0432434e62f4-kube-api-access-h7gcr\") pod \"perf-node-gather-daemonset-fk65t\" (UID: \"4622b91e-13d5-42f7-b636-0432434e62f4\") " pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.422846 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.422784    2543 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:44.560464 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.560372    2543 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"]
Apr 24 23:33:44.564520 ip-10-0-134-116 kubenswrapper[2543]: W0424 23:33:44.564476    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4622b91e_13d5_42f7_b636_0432434e62f4.slice/crio-c2816a73e058d8f1d4bf42ef34354e97573b946e39a75772c3fa3b3bb14df4e0 WatchSource:0}: Error finding container c2816a73e058d8f1d4bf42ef34354e97573b946e39a75772c3fa3b3bb14df4e0: Status 404 returned error can't find the container with id c2816a73e058d8f1d4bf42ef34354e97573b946e39a75772c3fa3b3bb14df4e0
Apr 24 23:33:44.685012 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.684957    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9gfb7_9639aef1-f7ec-48b0-90ee-98dfcfd3884a/dns/0.log"
Apr 24 23:33:44.704015 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.703992    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9gfb7_9639aef1-f7ec-48b0-90ee-98dfcfd3884a/kube-rbac-proxy/0.log"
Apr 24 23:33:44.762319 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.762285    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t" event={"ID":"4622b91e-13d5-42f7-b636-0432434e62f4","Type":"ContainerStarted","Data":"29c499dd9eaaac4d9a887511b0b3bd722705876afccd7fbdd4683fbed8d5076d"}
Apr 24 23:33:44.762437 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.762320    2543 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t" event={"ID":"4622b91e-13d5-42f7-b636-0432434e62f4","Type":"ContainerStarted","Data":"c2816a73e058d8f1d4bf42ef34354e97573b946e39a75772c3fa3b3bb14df4e0"}
Apr 24 23:33:44.766989 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.766970    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8tz2q_99e7bf73-8fe8-429f-b001-ed59f4e2bd01/dns-node-resolver/0.log"
Apr 24 23:33:44.776474 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:44.776375    2543 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t" podStartSLOduration=0.776333988 podStartE2EDuration="776.333988ms" podCreationTimestamp="2026-04-24 23:33:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-24 23:33:44.776070955 +0000 UTC m=+3828.202468486" watchObservedRunningTime="2026-04-24 23:33:44.776333988 +0000 UTC m=+3828.202731518"
Apr 24 23:33:45.205383 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:45.205352    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-879bc8cd9-gm7gg_11ba3c59-40dc-4673-ac40-6d4b1ff9bb3f/registry/0.log"
Apr 24 23:33:45.226616 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:45.226589    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2kt8q_bb485ff8-22b6-40e7-ab56-e64e94f03cbe/node-ca/0.log"
Apr 24 23:33:45.764809 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:45.764781    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:46.278016 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:46.277990    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bs42h_05906a72-1fb8-49fc-86c9-01d5fa92a7cd/serve-healthcheck-canary/0.log"
Apr 24 23:33:46.740273 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:46.740242    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lp9hc_19a6cd90-b20b-46e5-b7b5-f2e406db86b5/kube-rbac-proxy/0.log"
Apr 24 23:33:46.758730 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:46.758705    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lp9hc_19a6cd90-b20b-46e5-b7b5-f2e406db86b5/exporter/0.log"
Apr 24 23:33:46.778192 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:46.778167    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lp9hc_19a6cd90-b20b-46e5-b7b5-f2e406db86b5/extractor/0.log"
Apr 24 23:33:49.145493 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:49.145464    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-kw6zd_6c1bfbc3-0c19-4a0e-bb63-cfb27dcea163/s3-init/0.log"
Apr 24 23:33:49.166797 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:49.166767    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-djlqt_c210bd34-254e-4332-9936-f7a2a7ba46fe/s3-tls-init-custom/0.log"
Apr 24 23:33:49.188692 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:49.188674    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-k7cd4_f05004c7-2f8d-4ff1-a18e-eb07a140155a/s3-tls-init-serving/0.log"
Apr 24 23:33:49.212784 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:49.212760    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-86cc847c5c-88p77_2acbce06-e4b4-43e9-ae93-74c97e292076/seaweedfs/0.log"
Apr 24 23:33:51.777102 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:51.777059    2543 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rmwdq/perf-node-gather-daemonset-fk65t"
Apr 24 23:33:54.340210 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:54.340166    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-978tk_4369d13e-0dde-416a-b48e-2fec6e7a3317/kube-multus/0.log"
Apr 24 23:33:54.361605 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:54.361581    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2kcl_676d2ff5-547e-4f1e-a1ca-b87580102275/kube-multus-additional-cni-plugins/0.log"
Apr 24 23:33:54.381649 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:54.381605    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2kcl_676d2ff5-547e-4f1e-a1ca-b87580102275/egress-router-binary-copy/0.log"
Apr 24 23:33:54.401279 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:54.401259    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2kcl_676d2ff5-547e-4f1e-a1ca-b87580102275/cni-plugins/0.log"
Apr 24 23:33:54.422999 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:54.422984    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2kcl_676d2ff5-547e-4f1e-a1ca-b87580102275/bond-cni-plugin/0.log"
Apr 24 23:33:54.443265 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:54.443243    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2kcl_676d2ff5-547e-4f1e-a1ca-b87580102275/routeoverride-cni/0.log"
Apr 24 23:33:54.467030 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:54.467012    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2kcl_676d2ff5-547e-4f1e-a1ca-b87580102275/whereabouts-cni-bincopy/0.log"
Apr 24 23:33:54.520840 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:54.520820    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k2kcl_676d2ff5-547e-4f1e-a1ca-b87580102275/whereabouts-cni/0.log"
Apr 24 23:33:55.054705 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:55.054662    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pxq7f_df793786-616a-4001-9c32-2e4ebd101bb2/network-metrics-daemon/0.log"
Apr 24 23:33:55.079761 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:55.079714    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pxq7f_df793786-616a-4001-9c32-2e4ebd101bb2/kube-rbac-proxy/0.log"
Apr 24 23:33:56.192710 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:56.192639    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-controller/0.log"
Apr 24 23:33:56.213080 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:56.213053    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/0.log"
Apr 24 23:33:56.231915 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:56.231877    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovn-acl-logging/1.log"
Apr 24 23:33:56.250844 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:56.250816    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/kube-rbac-proxy-node/0.log"
Apr 24 23:33:56.277945 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:56.277922    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/kube-rbac-proxy-ovn-metrics/0.log"
Apr 24 23:33:56.303151 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:56.303130    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/northd/0.log"
Apr 24 23:33:56.333120 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:56.333094    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/nbdb/0.log"
Apr 24 23:33:56.356001 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:56.355978    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/sbdb/0.log"
Apr 24 23:33:56.478068 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:56.478021    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t72f2_8b7de9be-e366-4f28-8f32-0587de85e41b/ovnkube-controller/0.log"
Apr 24 23:33:57.723635 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:57.723612    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2lldp_e7d5adb7-ab2b-47df-859a-74f9de07e09b/network-check-target-container/0.log"
Apr 24 23:33:58.641920 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:58.641888    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9gkvt_db346dd5-9969-4a80-9cd0-5c8847bf7d6e/iptables-alerter/0.log"
Apr 24 23:33:59.322643 ip-10-0-134-116 kubenswrapper[2543]: I0424 23:33:59.322620    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-p8kkf_2fb76dd0-76d7-4eb0-ad36-56c371d98a7e/tuned/0.log"